-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathsave1
179 lines (178 loc) · 13.5 KB
/
save1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
amin@clusterNode1:~/mdfs/Hadoop-mdfs/hadoop-1.2.1$ bin/hadoop fs -rmr /user/amin/terasort-outputcleDeleted hdfs://clusterNode1.local:9000/user/amin/terasort-output
aamin@clusterNode1:~/mdfs/Hadoop-mdfs/hadoop-1.2.1$ clear
amin@clusterNode1:~/mdfs/Hadoop-mdfs/hadoop-1.2.1$ bin/hadoop jar hadoop-*examples*.jar terasort /user/amin/terasort-input /user/amin/terasort-output
13/09/27 17:45:43 INFO terasort.TeraSort: starting
13/09/27 17:45:43 INFO mapred.FileInputFormat: Total input paths to process : 2
13/09/27 17:45:43 ERROR mapred.FileInputFormat: Total Files 2
13/09/27 17:45:43 ERROR mapred.FileInputFormat: Total File length 1000000000
13/09/27 17:45:43 ERROR mapred.FileInputFormat: goalsize 500000000 numSplits 2 minSize 1
13/09/27 17:45:43 ERROR mapred.FileInputFormat: Block Size67108864 splitsize 67108864
13/09/27 17:45:43 ERROR mapred.FileInputFormat: File part-00000 Locations[Lorg.apache.hadoop.fs.BlockLocation;@765a16
13/09/27 17:45:44 ERROR mapred.FileInputFormat: Block Size67108864 splitsize 67108864
13/09/27 17:45:44 ERROR mapred.FileInputFormat: File part-00001 Locations[Lorg.apache.hadoop.fs.BlockLocation;@19d0a1
13/09/27 17:45:44 ERROR mapred.FileInputFormat: Total # of splits: 16
13/09/27 17:45:44 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/09/27 17:45:44 WARN snappy.LoadSnappy: Snappy native library not loaded
13/09/27 17:45:54 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
13/09/27 17:45:54 INFO compress.CodecPool: Got brand-new compressor
Making 1 from 100000 records
Step size is 100000.0
13/09/27 17:45:55 ERROR mapreduce.JobSubmissionFiles: real user amincurrentUseramin
13/09/27 17:45:55 ERROR mapred.JobClient: submitJobDir hdfs://clusterNode1.local:9000/home/amin/hadoop/mapred/staging/amin/.staging/job_201309271742_0001 staging area hdfs://clusterNode1.local:9000/home/amin/hadoop/mapred/staging/amin/.staging
13/09/27 17:45:55 ERROR mapred.JobClient: Creating splits at hdfs://clusterNode1.local:9000/home/amin/hadoop/mapred/staging/amin/.staging/job_201309271742_0001
13/09/27 17:45:55 ERROR mapred.JobClient: Writing Old Splits
13/09/27 17:45:55 INFO mapred.FileInputFormat: Total input paths to process : 2
13/09/27 17:45:55 ERROR mapred.FileInputFormat: Total Files 2
13/09/27 17:45:55 ERROR mapred.FileInputFormat: Total File length 1000000000
13/09/27 17:45:55 ERROR mapred.FileInputFormat: goalsize 500000000 numSplits 2 minSize 1
13/09/27 17:45:55 ERROR mapred.FileInputFormat: Block Size67108864 splitsize 67108864
13/09/27 17:45:55 ERROR mapred.FileInputFormat: File part-00000 Locations[Lorg.apache.hadoop.fs.BlockLocation;@617584
13/09/27 17:45:55 ERROR mapred.FileInputFormat: Block Size67108864 splitsize 67108864
13/09/27 17:45:55 ERROR mapred.FileInputFormat: File part-00001 Locations[Lorg.apache.hadoop.fs.BlockLocation;@ec1f9e
13/09/27 17:45:55 ERROR mapred.FileInputFormat: Total # of splits: 16
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 7Length 67108864Locations [clusterNode2, clusterNode3]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 126Length 67108864Locations [clusterNode3, clusterNode2]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 245Length 67108864Locations [clusterNode3, clusterNode2]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 364Length 67108864Locations [clusterNode3, clusterNode2]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 483Length 67108864Locations [clusterNode3, clusterNode2]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 602Length 67108864Locations [clusterNode2, clusterNode3]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 721Length 67108864Locations [clusterNode3, clusterNode2]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 840Length 67108864Locations [clusterNode3, clusterNode2]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 959Length 67108864Locations [clusterNode2, clusterNode3]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 1078Length 67108864Locations [clusterNode2, clusterNode3]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 1197Length 67108864Locations [clusterNode3, clusterNode2]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 1316Length 67108864Locations [clusterNode2, clusterNode3]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 1435Length 67108864Locations [clusterNode3, clusterNode2]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 1554Length 67108864Locations [clusterNode3, clusterNode2]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 1673Length 30237952Locations [clusterNode2, clusterNode3]
13/09/27 17:45:55 ERROR split.JobSplitWriter: class name org.apache.hadoop.mapred.FileSplitOffset 1792Length 30237952Locations [clusterNode2, clusterNode3]
13/09/27 17:45:57 INFO mapred.JobClient: Running job: job_201309271742_0001
13/09/27 17:45:58 INFO mapred.JobClient: map 0% reduce 0%
13/09/27 17:46:15 INFO mapred.JobClient: map 3% reduce 0%
13/09/27 17:46:16 INFO mapred.JobClient: map 6% reduce 0%
13/09/27 17:46:18 INFO mapred.JobClient: map 8% reduce 0%
13/09/27 17:46:19 INFO mapred.JobClient: map 10% reduce 0%
13/09/27 17:46:21 INFO mapred.JobClient: map 11% reduce 0%
13/09/27 17:46:22 INFO mapred.JobClient: map 16% reduce 0%
13/09/27 17:46:25 INFO mapred.JobClient: map 18% reduce 0%
13/09/27 17:46:35 INFO mapred.JobClient: map 21% reduce 0%
13/09/27 17:46:38 INFO mapred.JobClient: map 23% reduce 0%
13/09/27 17:46:40 INFO mapred.JobClient: map 29% reduce 0%
13/09/27 17:46:43 INFO mapred.JobClient: map 33% reduce 0%
13/09/27 17:46:47 INFO mapred.JobClient: map 35% reduce 2%
13/09/27 17:46:49 INFO mapred.JobClient: map 35% reduce 4%
13/09/27 17:46:50 INFO mapred.JobClient: map 36% reduce 4%
13/09/27 17:46:53 INFO mapred.JobClient: map 37% reduce 4%
13/09/27 17:47:08 INFO mapred.JobClient: map 37% reduce 8%
13/09/27 17:47:15 INFO mapred.JobClient: map 43% reduce 8%
13/09/27 17:47:17 INFO mapred.JobClient: map 46% reduce 8%
13/09/27 17:47:20 INFO mapred.JobClient: map 48% reduce 8%
13/09/27 17:47:21 INFO mapred.JobClient: map 51% reduce 8%
13/09/27 17:47:23 INFO mapred.JobClient: map 52% reduce 8%
13/09/27 17:47:24 INFO mapred.JobClient: map 54% reduce 8%
13/09/27 17:47:25 INFO mapred.JobClient: map 55% reduce 8%
13/09/27 17:47:28 INFO mapred.JobClient: map 56% reduce 8%
13/09/27 17:47:33 INFO mapred.JobClient: map 57% reduce 8%
13/09/27 17:47:34 INFO mapred.JobClient: map 58% reduce 8%
13/09/27 17:47:37 INFO mapred.JobClient: map 59% reduce 8%
13/09/27 17:47:39 INFO mapred.JobClient: map 59% reduce 10%
13/09/27 17:47:40 INFO mapred.JobClient: map 60% reduce 10%
13/09/27 17:47:42 INFO mapred.JobClient: map 60% reduce 12%
13/09/27 17:47:46 INFO mapred.JobClient: map 61% reduce 12%
13/09/27 17:47:51 INFO mapred.JobClient: map 62% reduce 12%
13/09/27 17:47:57 INFO mapred.JobClient: map 68% reduce 12%
13/09/27 17:48:02 INFO mapred.JobClient: map 69% reduce 12%
13/09/27 17:48:05 INFO mapred.JobClient: map 73% reduce 12%
13/09/27 17:48:08 INFO mapred.JobClient: map 74% reduce 12%
13/09/27 17:48:11 INFO mapred.JobClient: map 74% reduce 14%
13/09/27 17:48:23 INFO mapred.JobClient: map 76% reduce 14%
13/09/27 17:48:26 INFO mapred.JobClient: map 77% reduce 14%
13/09/27 17:48:27 INFO mapred.JobClient: map 79% reduce 14%
13/09/27 17:48:28 INFO mapred.JobClient: map 79% reduce 16%
13/09/27 17:48:31 INFO mapred.JobClient: map 80% reduce 18%
13/09/27 17:48:34 INFO mapred.JobClient: map 81% reduce 18%
13/09/27 17:48:37 INFO mapred.JobClient: map 82% reduce 18%
13/09/27 17:48:39 INFO mapred.JobClient: map 94% reduce 18%
13/09/27 17:48:40 INFO mapred.JobClient: map 96% reduce 18%
13/09/27 17:48:43 INFO mapred.JobClient: map 97% reduce 18%
13/09/27 17:48:46 INFO mapred.JobClient: map 98% reduce 18%
13/09/27 17:48:49 INFO mapred.JobClient: map 99% reduce 20%
13/09/27 17:48:52 INFO mapred.JobClient: map 99% reduce 22%
13/09/27 17:49:11 INFO mapred.JobClient: map 100% reduce 22%
13/09/27 17:49:17 INFO mapred.JobClient: map 100% reduce 27%
13/09/27 17:50:15 INFO mapred.JobClient: map 100% reduce 29%
13/09/27 17:51:20 INFO mapred.JobClient: map 100% reduce 31%
13/09/27 17:52:21 INFO mapred.JobClient: map 100% reduce 33%
13/09/27 17:52:55 INFO mapred.JobClient: map 100% reduce 66%
13/09/27 17:52:58 INFO mapred.JobClient: map 100% reduce 67%
13/09/27 17:53:04 INFO mapred.JobClient: map 100% reduce 68%
13/09/27 17:53:07 INFO mapred.JobClient: map 100% reduce 69%
13/09/27 17:53:10 INFO mapred.JobClient: map 100% reduce 70%
13/09/27 17:53:17 INFO mapred.JobClient: map 100% reduce 71%
13/09/27 17:53:19 INFO mapred.JobClient: map 100% reduce 72%
13/09/27 17:53:23 INFO mapred.JobClient: map 100% reduce 73%
13/09/27 17:53:29 INFO mapred.JobClient: map 100% reduce 74%
13/09/27 17:53:32 INFO mapred.JobClient: map 100% reduce 75%
13/09/27 17:53:35 INFO mapred.JobClient: map 100% reduce 76%
13/09/27 17:53:41 INFO mapred.JobClient: map 100% reduce 77%
13/09/27 17:53:44 INFO mapred.JobClient: map 100% reduce 78%
13/09/27 17:53:47 INFO mapred.JobClient: map 100% reduce 79%
13/09/27 17:53:53 INFO mapred.JobClient: map 100% reduce 80%
13/09/27 17:53:56 INFO mapred.JobClient: map 100% reduce 81%
13/09/27 17:54:03 INFO mapred.JobClient: map 100% reduce 82%
13/09/27 17:54:06 INFO mapred.JobClient: map 100% reduce 83%
13/09/27 17:54:09 INFO mapred.JobClient: map 100% reduce 84%
13/09/27 17:54:15 INFO mapred.JobClient: map 100% reduce 85%
13/09/27 17:54:18 INFO mapred.JobClient: map 100% reduce 86%
13/09/27 17:54:21 INFO mapred.JobClient: map 100% reduce 87%
13/09/27 17:54:27 INFO mapred.JobClient: map 100% reduce 88%
13/09/27 17:54:31 INFO mapred.JobClient: map 100% reduce 89%
13/09/27 17:54:33 INFO mapred.JobClient: map 100% reduce 90%
13/09/27 17:54:36 INFO mapred.JobClient: map 100% reduce 91%
13/09/27 17:54:43 INFO mapred.JobClient: map 100% reduce 92%
13/09/27 17:54:46 INFO mapred.JobClient: map 100% reduce 93%
13/09/27 17:54:49 INFO mapred.JobClient: map 100% reduce 94%
13/09/27 17:54:55 INFO mapred.JobClient: map 100% reduce 95%
13/09/27 17:54:58 INFO mapred.JobClient: map 100% reduce 96%
13/09/27 17:55:04 INFO mapred.JobClient: map 100% reduce 97%
13/09/27 17:55:07 INFO mapred.JobClient: map 100% reduce 98%
13/09/27 17:55:13 INFO mapred.JobClient: map 100% reduce 99%
13/09/27 17:55:17 INFO mapred.JobClient: map 100% reduce 100%
13/09/27 17:55:26 INFO mapred.JobClient: Job complete: job_201309271742_0001
13/09/27 17:55:26 INFO mapred.JobClient: Counters: 30
13/09/27 17:55:26 INFO mapred.JobClient: Job Counters
13/09/27 17:55:26 INFO mapred.JobClient: Launched reduce tasks=1
13/09/27 17:55:26 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=675704
13/09/27 17:55:26 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/09/27 17:55:26 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/09/27 17:55:26 INFO mapred.JobClient: Launched map tasks=17
13/09/27 17:55:26 INFO mapred.JobClient: Data-local map tasks=17
13/09/27 17:55:26 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=527646
13/09/27 17:55:26 INFO mapred.JobClient: File Input Format Counters
13/09/27 17:55:26 INFO mapred.JobClient: Bytes Read=1000057344
13/09/27 17:55:26 INFO mapred.JobClient: File Output Format Counters
13/09/27 17:55:26 INFO mapred.JobClient: Bytes Written=1000000000
13/09/27 17:55:26 INFO mapred.JobClient: FileSystemCounters
13/09/27 17:55:26 INFO mapred.JobClient: FILE_BYTES_READ=2382257412
13/09/27 17:55:26 INFO mapred.JobClient: HDFS_BYTES_READ=1000059248
13/09/27 17:55:26 INFO mapred.JobClient: FILE_BYTES_WRITTEN=3403211498
13/09/27 17:55:26 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1000000000
13/09/27 17:55:26 INFO mapred.JobClient: Map-Reduce Framework
13/09/27 17:55:26 INFO mapred.JobClient: Map output materialized bytes=1020000096
13/09/27 17:55:26 INFO mapred.JobClient: Map input records=10000000
13/09/27 17:55:26 INFO mapred.JobClient: Reduce shuffle bytes=1020000096
13/09/27 17:55:26 INFO mapred.JobClient: Spilled Records=33355441
13/09/27 17:55:26 INFO mapred.JobClient: Map output bytes=1000000000
13/09/27 17:55:26 INFO mapred.JobClient: Total committed heap usage (bytes)=2406342656
13/09/27 17:55:26 INFO mapred.JobClient: CPU time spent (ms)=693320
13/09/27 17:55:26 INFO mapred.JobClient: Map input bytes=1000000000
13/09/27 17:55:26 INFO mapred.JobClient: SPLIT_RAW_BYTES=1904
13/09/27 17:55:26 INFO mapred.JobClient: Combine input records=0
13/09/27 17:55:26 INFO mapred.JobClient: Reduce input records=10000000
13/09/27 17:55:26 INFO mapred.JobClient: Reduce input groups=10000000
13/09/27 17:55:26 INFO mapred.JobClient: Combine output records=0
13/09/27 17:55:26 INFO mapred.JobClient: Physical memory (bytes) snapshot=2708475904
13/09/27 17:55:26 INFO mapred.JobClient: Reduce output records=10000000
13/09/27 17:55:26 INFO mapred.JobClient: Virtual memory (bytes) snapshot=6441357312
13/09/27 17:55:26 INFO mapred.JobClient: Map output records=10000000
13/09/27 17:55:26 INFO terasort.TeraSort: done