Hi,

I have two files file1 an file2. I want to take each line from two files and compare with a keyword if the keyword matches in both the files. The complete line which matches from file1 should be written to file2 at the position below the matching line.

File1 content:

(Ingestion.py) test_Execution_Status_Update[Invalid_Hdfs_Log_Path_Test_Case]
Moving log file from local temp '/workspace/ws_facilities_datalake/archibus/pytest_test/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log' of driver host 'ddlhdpdev08' to hdfs '/hivestore/dev/workspace/ws_facilities/archibus/bl1/stg_test1/'
Log file move from local temp '/workspace/ws_facilities_datalake/archibus/pytest_test/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log' of driver host 'ddlhdpdev08' to hdfs '/hivestore/dev/workspace/ws_facilities/archibus/bl1/stg_test1/' failed and the error is Traceback (most recent call last):
 File "/workspace/ws_facilities_datalake/archibus/pytest_test/Ingestion/Ingestion.py", line 98, in Execution_Status_Update
  subprocess.check_output('hadoop fs -moveFromLocal -f {} {}'.format(Log_File,Hdfs_Log_Path), shell=True)
 File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
  raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'hadoop fs -moveFromLocal -f /workspace/ws_facilities_datalake/archibus/pytest_test/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log /hivestore/dev/workspace/ws_facilities/archibus/bl1/stg_test1/' returned non-zero exit status 1


(Ingestion.py) test_Execution_Status_Update[Valid_Hdfs_Log_Path_Test_Case0]
Moving log file from local temp '/workspace/ws_facilities_datalake/archibus/pytest_test/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2022.log' of driver host 'ddlhdpdev08' to hdfs '/hivestore/dev/workspace/ws_facilities/archibus/bl/stg_test'
Log file move from local temp to hdfs is successful

File2 Content:

2021-01-06 02:59:46 (Ingestion.py:107) test_Execution_Status_Update[Invalid_Hdfs_Log_Path_Test_Case] [ERROR] Log file move from local temp '/workspace/ws_facilities_datalake/archibus/pytest/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log' of driver host 'ddlhdpdev08' to hdfs '/hivestore/dev/workspace/ws_facilities/archibus/bl1/stg_test1/' failed and the error is Traceback (most recent call last):
 File "/workspace/ws_facilities_datalake/archibus/pytest/Ingestion/Ingestion.py", line 98, in Execution_Status_Update
  subprocess.check_output('hadoop fs -moveFromLocal -f {} {}'.format(Log_File,Hdfs_Log_Path), shell=True)
 File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
  raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'hadoop fs -moveFromLocal -f /workspace/ws_facilities_datalake/archibus/pytest/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log /hivestore/dev/workspace/ws_facilities/archibus/bl1/stg_test1/' returned non-zero exit status 1

2021-01-06 02:59:50 (Ingestion.py:116) test_Source_Password_Decryption[Password_Decryption_Failure_Test_Case] [INFO] Reading password key file '/workspace/ws_facilities_datalake/archibus/pytest/Ingestion/archibus_user_dev.key'
2021-01-06 02:59:50 (Ingestion.py:121) test_Source_Password_Decryption[Password_Decryption_Failure_Test_Case] [INFO] Password key file read is successful
2021-01-06 02:59:50 (Ingestion.py:122) test_Source_Password_Decryption[Password_Decryption_Failure_Test_Case] [INFO]

Here in my case the keyword that should match is:

"test_Execution_Status_Update[Invalid_Hdfs_Log_Path_Test_Case] "

Output Expected: File2

2021-01-05 04:45:39 (Ingestion.py:107) test_Execution_Status_Update[Invalid_Hdfs_Log_Path_Test_Case] [ERROR] Log file move from local temp '/workspace/ws_facilities_datalake/archibus/pytest/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log' of driver host 'ddlhdpdev08' to hdfs '/hivestore/dev/workspace/ws_facilities/archibus/bl1/stg_test1/' failed and the error is Traceback (most recent call last):
 File "/workspace/ws_facilities_datalake/archibus/pytest/Ingestion/Ingestion.py", line 98, in Execution_Status_Update
  subprocess.check_output('hadoop fs -moveFromLocal -f {} {}'.format(Log_File,Hdfs_Log_Path), shell=True)
 File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
  raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'hadoop fs -moveFromLocal -f /workspace/ws_facilities_datalake/archibus/pytest/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log /hivestore/dev/workspace/ws_facilities/archibus/bl1/stg_test1/' returned non-zero exit status 1
(Ingestion.py) test_Execution_Status_Update[Invalid_Hdfs_Log_Path_Test_Case]
Moving log file from local temp '/workspace/ws_facilities_datalake/archibus/pytest_test/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log' of driver host 'ddlhdpdev08' to hdfs '/hivestore/dev/workspace/ws_facilities/archibus/bl1/stg_test1/'
Log file move from local temp '/workspace/ws_facilities_datalake/archibus/pytest_test/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log' of driver host 'ddlhdpdev08' to hdfs '/hivestore/dev/workspace/ws_facilities/archibus/bl1/stg_test1/' failed and the error is Traceback (most recent call last):
 File "/workspace/ws_facilities_datalake/archibus/pytest_test/Ingestion/Ingestion.py", line 98, in Execution_Status_Update
  subprocess.check_output('hadoop fs -moveFromLocal -f {} {}'.format(Log_File,Hdfs_Log_Path), shell=True)
 File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
  raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'hadoop fs -moveFromLocal -f /workspace/ws_facilities_datalake/archibus/pytest_test/Ingestion/m_AB_TO_STG_bl_data_Ingestion_2024.log

2021-01-05 04:45:42 (Ingestion.py:116) test_Source_Password_Decryption[Password_Decryption_Function_Failure_Test_Case] [INFO] Reading password key file '/workspace/ws_facilities_datalake/archibus/pytest/Ingestion/archibus_user_dev.key'
2021-01-05 04:45:42 (Ingestion.py:121) test_Source_Password_Decryption[Password_Decryption_Function_Failure_Test_Case] [INFO] Password key file read is successful
2021-01-05 04:45:42 (Ingestion.py:122) test_Source_Password_Decryption[Password_Decryption_Function_Failure_Test_Case] [INFO]
2021-01-05 04:45:42 (Ingestion.py:129) test_Source_Password_Decryption[Password_Decryption_Function_Failure_Test_Case] [INFO] Reading encrypted password file '/workspace/ws_facilities_datalake/archibus/pytest/Ingestion/archibus_user_password_dev.encrypted'
2021-01-05 04:45:42 (Ingestion.py:134) test_Source_Password_Decryption[Password_Decryption_Function_Failure_Test_Case] [INFO] Encrypted password file read is successful