Χαίρετε,
Προσπαθώντας να τρέξω δικό μου java αρχείο ή από τα hadoop examples το πρόγραμμα κολλάει αρχικά και δεν προχωράει καθόλου. root@master:~# yarn jar hadoop-mapreduce-examples-2.6.3.jar wordcount /input/enwiki-latest-all-titles-in-ns0.1000.txt /output/wordcount.txt INFO client.RMProxy: Connecting to ResourceManager at master/ 192.168.1.1:8050 INFO input.FileInputFormat: Total input paths to process : 1 INFO mapreduce.JobSubmitter: number of splits:1 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489339353440_0002 INFO impl.YarnClientImpl: Submitted application application_1489339353440_0002 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1489339353440_0002/ INFO mapreduce.Job: Running job: job_1489339353440_0002
Κάποιες φορές προχωράει μια γραμμή παραπάνω: INFO mapreduce.Job: map 0% reduce 0% και κολλάει εδώ.
Tόσο στον master όσο και τον slave μια χαρά τρέχει το hdfs και το yarn πριν την εκτέλεση, root@master:~# jps 15066 SecondaryNameNode 14922 NameNode 15239 ResourceManager 15500 Jps
root@slave:~# jps 2239 NodeManager 2142 DataNode 2367 Jps
όσο και κατα την διάρκεια της εκτέλεσης. root@master:~# jps 15066 SecondaryNameNode 15544 RunJar 14922 NameNode 15239 ResourceManager 15637 Jps
root@slave:~# jps 2446 MRAppMaster 2239 NodeManager 2549 Jps 2142 DataNode
Ψάχνοντας στο internet μάλλον το θέμα είναι είτε ότι ο yarn έχει μικρή μνήμη ανα mapper αλλά αυτό δεν είναι λογικό καθώς κάθε block είναι λίγο μικρότερο από 32mB ενώ δίνουμε στο node 1800mB ή γενικότερα επειδή το τρέχουμε σε VM και κολλάει για αυτό.
Ευχαριστώ πολύ Γιώργος Αγγελόπουλος
Γειά σου Γιώργο,
κατά την ώρα που "κολλάει" το Map Reduce job, τι log παίρνεις; Από ότι φαίνεται από το output που επισυνάπτεις, το job έχει γίνει spawn (το RunJar είναι το binary που σηκώνεται και τρέχει το Jar σου και ο MRAppMaster είναι ο Application Master) αλλά δεν έχει γίνει spawn κάποιο container που θα τρέξει το actual job. Υπάρχει αρκετή διαθέσιμη μνήμη για να σηκωθεί ο container; Το UI του YARN σου δίνει κάποια παραπάνω πληροφορία;
Δες τα logs του NodeManager (που είναι ο υπεύθυνος να σηκώσει και τον container) να δούμε για ποιό λόγο κολλάει.
Καλή συνέχεια, Γιάννης
2017-03-12 21:50 GMT+02:00 Geo Angelopoulos geo.angelotti@gmail.com:
Χαίρετε,
Προσπαθώντας να τρέξω δικό μου java αρχείο ή από τα hadoop examples το πρόγραμμα κολλάει αρχικά και δεν προχωράει καθόλου. root@master:~# yarn jar hadoop-mapreduce-examples-2.6.3.jar wordcount /input/enwiki-latest-all-titles-in-ns0.1000.txt /output/wordcount.txt INFO client.RMProxy: Connecting to ResourceManager at master/ 192.168.1.1:8050 INFO input.FileInputFormat: Total input paths to process : 1 INFO mapreduce.JobSubmitter: number of splits:1 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489339353440_0002 INFO impl.YarnClientImpl: Submitted application application_1489339353440_0002 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/ application_1489339353440_0002/ INFO mapreduce.Job: Running job: job_1489339353440_0002
Κάποιες φορές προχωράει μια γραμμή παραπάνω: INFO mapreduce.Job: map 0% reduce 0% και κολλάει εδώ.
Tόσο στον master όσο και τον slave μια χαρά τρέχει το hdfs και το yarn πριν την εκτέλεση, root@master:~# jps 15066 SecondaryNameNode 14922 NameNode 15239 ResourceManager 15500 Jps
root@slave:~# jps 2239 NodeManager 2142 DataNode 2367 Jps
όσο και κατα την διάρκεια της εκτέλεσης. root@master:~# jps 15066 SecondaryNameNode 15544 RunJar 14922 NameNode 15239 ResourceManager 15637 Jps
root@slave:~# jps 2446 MRAppMaster 2239 NodeManager 2549 Jps 2142 DataNode
Ψάχνοντας στο internet μάλλον το θέμα είναι είτε ότι ο yarn έχει μικρή μνήμη ανα mapper αλλά αυτό δεν είναι λογικό καθώς κάθε block είναι λίγο μικρότερο από 32mB ενώ δίνουμε στο node 1800mB ή γενικότερα επειδή το τρέχουμε σε VM και κολλάει για αυτό.
Ευχαριστώ πολύ Γιώργος Αγγελόπουλος
Atds mailing list Atds@lists.cslab.ece.ntua.gr http://lists.cslab.ece.ntua.gr/mailman/listinfo/atds
Γεια σου Γιάννη, το log του resource manager είναι το ακόλουθο:
2017-03-13 17:29:01,619 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 3 2017-03-13 17:29:04,343 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1489350810171_0003 2017-03-13 17:29:04,344 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1489350810171_0003 State change from NEW to NEW_SAVING 2017-03-13 17:29:04,344 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1489350810171_0003 2017-03-13 17:29:04,344 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1489350810171_0003 State change from NEW_SAVING to SUBMITTED 2017-03-13 17:29:04,344 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1489350810171_0003 user: root leaf-queue of parent: root #applications: 3 2017-03-13 17:29:04,345 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1489350810171_0003 from user: root, in queue: default 2017-03-13 17:29:04,345 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1489350810171_0003 State change from SUBMITTED to ACCEPTED 2017-03-13 17:29:04,345 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1489350810171_0003_000001 2017-03-13 17:29:04,346 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1489350810171_0003_000001 State change from NEW to SUBMITTED 2017-03-13 17:29:04,346 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 3 submitted by user root 2017-03-13 17:29:04,346 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root IP=83.212.97.127 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1489350810171_0003 2017-03-13 17:29:04,347 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not starting application as amIfStarted exceeds amLimit 2017-03-13 17:29:04,347 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not starting application as amIfStarted exceeds amLimit 2017-03-13 17:29:04,347 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1489350810171_0003 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@37342445, leaf-queue: default #user-pending-applications: 2 #user-active-applications: 1 #queue-pending-applications: 2 #queue-active-applications: 1 2017-03-13 17:29:04,347 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1489350810171_0003_000001 to scheduler from user root in queue default 2017-03-13 17:29:04,348 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1489350810171_0003_000001 State change from SUBMITTED to SCHEDULED
του master namenode είναι:
2017-03-13 17:29:02,149 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 40 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 0 Number of syncs: 31 SyncTimes(ms): 82 2017-03-13 17:29:02,361 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.jar. BP-1314251074-192.168.1.1-1479317663787 blk_1073741921_1098{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} 2017-03-13 17:29:02,816 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.1.2:50010 is added to blk_1073741921_1098{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} size 0 2017-03-13 17:29:02,834 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.jar is closed by DFSClient_NONMAPREDUCE_-429767746_1 2017-03-13 17:29:02,856 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Increasing replication from 1 to 10 for /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.jar 2017-03-13 17:29:03,046 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Increasing replication from 1 to 10 for /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.split 2017-03-13 17:29:03,063 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.split. BP-1314251074-192.168.1.1-1479317663787 blk_1073741922_1099{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} 2017-03-13 17:29:03,084 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.1.2:50010 is added to blk_1073741922_1099{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} size 0 2017-03-13 17:29:03,089 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.split is closed by DFSClient_NONMAPREDUCE_-429767746_1 2017-03-13 17:29:03,108 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.splitmetainfo. BP-1314251074-192.168.1.1-1479317663787 blk_1073741923_1100{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} 2017-03-13 17:29:03,126 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.1.2:50010 is added to blk_1073741923_1100{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} size 0 2017-03-13 17:29:03,131 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.splitmetainfo is closed by DFSClient_NONMAPREDUCE_-429767746_1 2017-03-13 17:29:03,719 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.xml. BP-1314251074-192.168.1.1-1479317663787 blk_1073741924_1101{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} 2017-03-13 17:29:03,789 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.1.2:50010 is added to blk_1073741924_1101{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} size 0 2017-03-13 17:29:03,796 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.xml is closed by DFSClient_NONMAPREDUCE_-429767746_1 2017-03-13 17:29:16,533 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:29:16,536 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:29:46,534 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:29:46,537 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:30:16,534 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:30:16,537 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:30:46,535 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:30:46,538 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:31:16,536 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:31:16,538 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 2 millisecond(s). 2017-03-13 17:31:46,537 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:31:46,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:32:16,537 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:32:16,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:32:46,538 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:32:46,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 2 millisecond(s). 2017-03-13 17:33:16,539 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:33:16,542 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:33:46,539 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:33:46,542 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:34:16,539 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:34:16,542 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:34:39,337 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 83.212.97.127 2017-03-13 17:34:39,338 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 2017-03-13 17:34:39,338 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 334550 2017-03-13 17:34:39,339 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 68 Total time for transactions(ms): 10 Number of transactions batched in Syncs: 0 Number of syncs: 51 SyncTimes(ms): 119 2017-03-13 17:34:39,341 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 68 Total time for transactions(ms): 10 Number of transactions batched in Syncs: 0 Number of syncs: 52 SyncTimes(ms): 121 2017-03-13 17:34:39,342 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /opt/hdfsnames/current/edits_inprogress_0000000000000334550 -> /opt/hdfsnames/current/edits_0000000000000334550-0000000000000334617 2017-03-13 17:34:39,343 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 334618 2017-03-13 17:34:39,843 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.01s at 1125.00 KB/s 2017-03-13 17:34:39,844 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000334617 size 9385 bytes. 2017-03-13 17:34:39,895 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 334549 2017-03-13 17:34:39,896 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old image FSImageFile(file=/opt/hdfsnames/current/fsimage_0000000000000334547, cpktTxId=0000000000000334547) 2017-03-13 17:34:40,203 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Purging logs older than 314005 2017-03-13 17:34:40,267 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old edit log EditLogFile(file=/opt/hdfsnames/current/edits_0000000000000314003-0000000000000314004,first=0000000000000314003,last=0000000000000314004,inProgress=false,hasCorruptHeader=false) 2017-03-13 17:34:46,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:34:46,543 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:35:16,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:35:16,543 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:35:20,865 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: there are no corrupt file blocks. 2017-03-13 17:35:30,625 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: there are no corrupt file blocks. 2017-03-13 17:35:46,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:35:46,543 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:36:16,541 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:36:16,544 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s).
και του secondarynamenode δεν γράφει τίποτα καινούργιο.
Δεν με συνδέει στο http://master:8088 του yarn.
Ευχαριστώ πολύ
Γιώργος Αγγελόπουλος
Στις 13 Μαρτίου 2017 - 4:34 μ.μ., ο χρήστης Giannis Giannakopoulos ggian@cslab.ece.ntua.gr έγραψε:
Γειά σου Γιώργο,
κατά την ώρα που "κολλάει" το Map Reduce job, τι log παίρνεις; Από ότι φαίνεται από το output που επισυνάπτεις, το job έχει γίνει spawn (το RunJar είναι το binary που σηκώνεται και τρέχει το Jar σου και ο MRAppMaster είναι ο Application Master) αλλά δεν έχει γίνει spawn κάποιο container που θα τρέξει το actual job. Υπάρχει αρκετή διαθέσιμη μνήμη για να σηκωθεί ο container; Το UI του YARN σου δίνει κάποια παραπάνω πληροφορία;
Δες τα logs του NodeManager (που είναι ο υπεύθυνος να σηκώσει και τον container) να δούμε για ποιό λόγο κολλάει.
Καλή συνέχεια, Γιάννης
2017-03-12 21:50 GMT+02:00 Geo Angelopoulos geo.angelotti@gmail.com:
Χαίρετε,
Προσπαθώντας να τρέξω δικό μου java αρχείο ή από τα hadoop examples το πρόγραμμα κολλάει αρχικά και δεν προχωράει καθόλου. root@master:~# yarn jar hadoop-mapreduce-examples-2.6.3.jar wordcount /input/enwiki-latest-all-titles-in-ns0.1000.txt /output/wordcount.txt INFO client.RMProxy: Connecting to ResourceManager at master/ 192.168.1.1:8050 INFO input.FileInputFormat: Total input paths to process : 1 INFO mapreduce.JobSubmitter: number of splits:1 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489339353440_0002 INFO impl.YarnClientImpl: Submitted application application_1489339353440_0002 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1489339353440_0002/ INFO mapreduce.Job: Running job: job_1489339353440_0002
Κάποιες φορές προχωράει μια γραμμή παραπάνω: INFO mapreduce.Job: map 0% reduce 0% και κολλάει εδώ.
Tόσο στον master όσο και τον slave μια χαρά τρέχει το hdfs και το yarn πριν την εκτέλεση, root@master:~# jps 15066 SecondaryNameNode 14922 NameNode 15239 ResourceManager 15500 Jps
root@slave:~# jps 2239 NodeManager 2142 DataNode 2367 Jps
όσο και κατα την διάρκεια της εκτέλεσης. root@master:~# jps 15066 SecondaryNameNode 15544 RunJar 14922 NameNode 15239 ResourceManager 15637 Jps
root@slave:~# jps 2446 MRAppMaster 2239 NodeManager 2549 Jps 2142 DataNode
Ψάχνοντας στο internet μάλλον το θέμα είναι είτε ότι ο yarn έχει μικρή μνήμη ανα mapper αλλά αυτό δεν είναι λογικό καθώς κάθε block είναι λίγο μικρότερο από 32mB ενώ δίνουμε στο node 1800mB ή γενικότερα επειδή το τρέχουμε σε VM και κολλάει για αυτό.
Ευχαριστώ πολύ Γιώργος Αγγελόπουλος
Atds mailing list Atds@lists.cslab.ece.ntua.gr http://lists.cslab.ece.ntua.gr/mailman/listinfo/atds
--
Ioannis Giannakopoulos PhD Candidate, Systems Administrator Computing Systems Laboratory, School of ECE National Technical University of Athens e-mail: ggian@cslab.ece.ntua.gr site: http://www.cslab.ece.ntua.gr/~ggian/ skype name: giagiannis tel: +302107721530 <+30%2021%200772%201530>
Γειά σου Γιώργο, οκ, κάτι που φαίνεται να έχει ενδιαφέρον είναι το εξής:
2017-03-13 17:29:04,347 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not starting application as amIfStarted exceeds amLimit
για το οποίο βρίσκω αυτό http://stackoverflow.com/questions/33465300/why-does-yarn-job-not-transition-to-running-state. Κοινώς, το cluster σου δεν έχει αρκετά resources για να σηκώσει το child container (που θα τρέξει το job) + τον container που κάνει orchestrate το job. Από το προηγούμενο email σου φαίνεται πως έχεις μόνο έναν NodeManager, άρα σε ένα μόνο μηχάνημα μπορείς να σηκώσεις child containers. Σου προτείνω να προσθέσεις και τον master node στους slaves, προσθέτοντας αντίστοιχα το entry "master" στο slaves file στον HADOOP_CONF_DIR και να ξαναδοκιμάσεις (αυτό προτείνεται και στον οδηγό που έχει αναρτηθεί στο mycourses).
Τέλος, και με αφορμή το παραπάνω, θέλω να υπερτονίσω τη σημασία των log files κατά το debugging. Είναι απαραίτητο όταν αντιμετωπίζουμε μια απρόβλεπτη/προβληματική συμπεριφορά να εξετάζουμε τα log files των services από τα οποία εξαρτώμαστε καθώς *και* του λειτουργικού συστήματος (αν τα πρώτα δεν αρκούν), σεβόμενοι τη *λογική σειρά με την οποια πρέπει να εκτελεστούν τα πράγματα.* Πχ, εδώ ενώ το job ξεκινάει, δεν σηκώνεται o child container, άρα κάνουμε τις εξής ερωτήσεις: Ο NodeManager ζει (χρήσιμες εντολές: jps ή ps -ef | grep java); Παρατηρώ κάτι περίεργο στο log file του; Αν όλα είναι καλά με αυτό, ο ResourceManager λειτουργεί κανονικά; Αν ναι, το HDFS είναι προσπελάσιμο (Namenode πρωτίστως και DataNodes); Μήπως έχω corrupted αρχεία; Αν όχι, μήπως υπάρχει δικτυακό πρόβλημα (χρησιμες εντολές: ping, traceroute, ip); Ακολουθώντας ένα τέτοιο workflow θα μπορούμε να αντιμετωπίζουμε ό,τι πρόβλημα παρουσιαστεί :).
Καλή συνέχεια,
Γ.
On Mon, Mar 13, 2017 at 6:12 PM, Geo Angelopoulos geo.angelotti@gmail.com wrote:
Γεια σου Γιάννη, το log του resource manager είναι το ακόλουθο:
2017-03-13 17:29:01,619 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 3 2017-03-13 17:29:04,343 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1489350810171_0003 2017-03-13 17:29:04,344 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1489350810171_0003 State change from NEW to NEW_SAVING 2017-03-13 17:29:04,344 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1489350810171_0003 2017-03-13 17:29:04,344 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1489350810171_0003 State change from NEW_SAVING to SUBMITTED 2017-03-13 17:29:04,344 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1489350810171_0003 user: root leaf-queue of parent: root #applications: 3 2017-03-13 17:29:04,345 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1489350810171_0003 from user: root, in queue: default 2017-03-13 17:29:04,345 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1489350810171_0003 State change from SUBMITTED to ACCEPTED 2017-03-13 17:29:04,345 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1489350810171_0003_000001 2017-03-13 17:29:04,346 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1489350810171_0003_000001 State change from NEW to SUBMITTED 2017-03-13 17:29:04,346 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 3 submitted by user root 2017-03-13 17:29:04,346 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root IP=83.212.97.127 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1489350810171_0003 2017-03-13 17:29:04,347 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not starting application as amIfStarted exceeds amLimit 2017-03-13 17:29:04,347 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not starting application as amIfStarted exceeds amLimit 2017-03-13 17:29:04,347 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1489350810171_0003 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@37342445, leaf-queue: default #user-pending-applications: 2 #user-active-applications: 1 #queue-pending-applications: 2 #queue-active-applications: 1 2017-03-13 17:29:04,347 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1489350810171_0003_000001 to scheduler from user root in queue default 2017-03-13 17:29:04,348 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1489350810171_0003_000001 State change from SUBMITTED to SCHEDULED
του master namenode είναι:
2017-03-13 17:29:02,149 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 40 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 0 Number of syncs: 31 SyncTimes(ms): 82 2017-03-13 17:29:02,361 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.jar. BP-1314251074-192.168.1.1-1479317663787 blk_1073741921_1098{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} 2017-03-13 17:29:02,816 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.1.2:50010 is added to blk_1073741921_1098{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} size 0 2017-03-13 17:29:02,834 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.jar is closed by DFSClient_NONMAPREDUCE_-429767746_1 2017-03-13 17:29:02,856 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Increasing replication from 1 to 10 for /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.jar 2017-03-13 17:29:03,046 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Increasing replication from 1 to 10 for /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.split 2017-03-13 17:29:03,063 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.split. BP-1314251074-192.168.1.1-1479317663787 blk_1073741922_1099{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} 2017-03-13 17:29:03,084 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.1.2:50010 is added to blk_1073741922_1099{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} size 0 2017-03-13 17:29:03,089 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.split is closed by DFSClient_NONMAPREDUCE_-429767746_1 2017-03-13 17:29:03,108 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.splitmetainfo. BP-1314251074-192.168.1.1-1479317663787 blk_1073741923_1100{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} 2017-03-13 17:29:03,126 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.1.2:50010 is added to blk_1073741923_1100{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} size 0 2017-03-13 17:29:03,131 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.splitmetainfo is closed by DFSClient_NONMAPREDUCE_-429767746_1 2017-03-13 17:29:03,719 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.xml. BP-1314251074-192.168.1.1-1479317663787 blk_1073741924_1101{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} 2017-03-13 17:29:03,789 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 192.168.1.2:50010 is added to blk_1073741924_1101{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-dbe63150-5003-43dd-9d08-de7d8724ef9a:NORMAL:192.168.1.2:50010|RBW]]} size 0 2017-03-13 17:29:03,796 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1489350810171_0003/job.xml is closed by DFSClient_NONMAPREDUCE_-429767746_1 2017-03-13 17:29:16,533 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:29:16,536 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:29:46,534 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:29:46,537 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:30:16,534 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:30:16,537 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:30:46,535 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:30:46,538 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:31:16,536 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:31:16,538 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 2 millisecond(s). 2017-03-13 17:31:46,537 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:31:46,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:32:16,537 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:32:16,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:32:46,538 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:32:46,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 2 millisecond(s). 2017-03-13 17:33:16,539 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:33:16,542 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:33:46,539 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:33:46,542 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:34:16,539 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:34:16,542 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:34:39,337 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 83.212.97.127 2017-03-13 17:34:39,338 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 2017-03-13 17:34:39,338 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 334550 2017-03-13 17:34:39,339 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 68 Total time for transactions(ms): 10 Number of transactions batched in Syncs: 0 Number of syncs: 51 SyncTimes(ms): 119 2017-03-13 17:34:39,341 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 68 Total time for transactions(ms): 10 Number of transactions batched in Syncs: 0 Number of syncs: 52 SyncTimes(ms): 121 2017-03-13 17:34:39,342 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /opt/hdfsnames/current/edits_inprogress_0000000000000334550 -> /opt/hdfsnames/current/edits_0000000000000334550-0000000000000334617 2017-03-13 17:34:39,343 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 334618 2017-03-13 17:34:39,843 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.01s at 1125.00 KB/s 2017-03-13 17:34:39,844 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000334617 size 9385 bytes. 2017-03-13 17:34:39,895 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 334549 2017-03-13 17:34:39,896 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old image FSImageFile(file=/opt/hdfsnames/current/fsimage_0000000000000334547, cpktTxId=0000000000000334547) 2017-03-13 17:34:40,203 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Purging logs older than 314005 2017-03-13 17:34:40,267 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old edit log EditLogFile(file=/opt/hdfsnames/current/edits_0000000000000314003-0000000000000314004,first=0000000000000314003,last=0000000000000314004,inProgress=false,hasCorruptHeader=false) 2017-03-13 17:34:46,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:34:46,543 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:35:16,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:35:16,543 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:35:20,865 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: there are no corrupt file blocks. 2017-03-13 17:35:30,625 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: there are no corrupt file blocks. 2017-03-13 17:35:46,540 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-03-13 17:35:46,543 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s). 2017-03-13 17:36:16,541 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-03-13 17:36:16,544 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 3 millisecond(s).
και του secondarynamenode δεν γράφει τίποτα καινούργιο.
Δεν με συνδέει στο http://master:8088 του yarn.
Ευχαριστώ πολύ
Γιώργος Αγγελόπουλος
Στις 13 Μαρτίου 2017 - 4:34 μ.μ., ο χρήστης Giannis Giannakopoulos ggian@cslab.ece.ntua.gr έγραψε:
Γειά σου Γιώργο,
κατά την ώρα που "κολλάει" το Map Reduce job, τι log παίρνεις; Από ότι φαίνεται από το output που επισυνάπτεις, το job έχει γίνει spawn (το RunJar είναι το binary που σηκώνεται και τρέχει το Jar σου και ο MRAppMaster είναι ο Application Master) αλλά δεν έχει γίνει spawn κάποιο container που θα τρέξει το actual job. Υπάρχει αρκετή διαθέσιμη μνήμη για να σηκωθεί ο container; Το UI του YARN σου δίνει κάποια παραπάνω πληροφορία;
Δες τα logs του NodeManager (που είναι ο υπεύθυνος να σηκώσει και τον container) να δούμε για ποιό λόγο κολλάει.
Καλή συνέχεια, Γιάννης
2017-03-12 21:50 GMT+02:00 Geo Angelopoulos geo.angelotti@gmail.com:
Χαίρετε,
Προσπαθώντας να τρέξω δικό μου java αρχείο ή από τα hadoop examples το πρόγραμμα κολλάει αρχικά και δεν προχωράει καθόλου. root@master:~# yarn jar hadoop-mapreduce-examples-2.6.3.jar wordcount /input/enwiki-latest-all-titles-in-ns0.1000.txt /output/wordcount.txt INFO client.RMProxy: Connecting to ResourceManager at master/ 192.168.1.1:8050 INFO input.FileInputFormat: Total input paths to process : 1 INFO mapreduce.JobSubmitter: number of splits:1 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489339353440_0002 INFO impl.YarnClientImpl: Submitted application application_1489339353440_0002 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1489339353440_0002/ INFO mapreduce.Job: Running job: job_1489339353440_0002
Κάποιες φορές προχωράει μια γραμμή παραπάνω: INFO mapreduce.Job: map 0% reduce 0% και κολλάει εδώ.
Tόσο στον master όσο και τον slave μια χαρά τρέχει το hdfs και το yarn πριν την εκτέλεση, root@master:~# jps 15066 SecondaryNameNode 14922 NameNode 15239 ResourceManager 15500 Jps
root@slave:~# jps 2239 NodeManager 2142 DataNode 2367 Jps
όσο και κατα την διάρκεια της εκτέλεσης. root@master:~# jps 15066 SecondaryNameNode 15544 RunJar 14922 NameNode 15239 ResourceManager 15637 Jps
root@slave:~# jps 2446 MRAppMaster 2239 NodeManager 2549 Jps 2142 DataNode
Ψάχνοντας στο internet μάλλον το θέμα είναι είτε ότι ο yarn έχει μικρή μνήμη ανα mapper αλλά αυτό δεν είναι λογικό καθώς κάθε block είναι λίγο μικρότερο από 32mB ενώ δίνουμε στο node 1800mB ή γενικότερα επειδή το τρέχουμε σε VM και κολλάει για αυτό.
Ευχαριστώ πολύ Γιώργος Αγγελόπουλος
Atds mailing list Atds@lists.cslab.ece.ntua.gr http://lists.cslab.ece.ntua.gr/mailman/listinfo/atds
--
Ioannis Giannakopoulos PhD Candidate, Systems Administrator Computing Systems Laboratory, School of ECE National Technical University of Athens e-mail: ggian@cslab.ece.ntua.gr site: http://www.cslab.ece.ntua.gr/~ggian/ skype name: giagiannis tel: +302107721530 <+30%2021%200772%201530>
-
Geo Angelopoulos -
Giannis Giannakopoulos