How to split a log file into several csv files with python

Published on Author Code Father

Looking at the code there’s a few things that are possibly wrong:

the csv reader should take a file handler, not a single line.
the reader delimiter should not be a single space character as it looks like the actual delimiter in your logs is a variable number of multiple space characters.
the looping logic seems to be a bit off, confusing files/lines/rows a bit.
You may be looking at something like the code below (pending clarification in the question):

import csv
NEW_LOG_DELIMITER = ‘MYLOG’

def write_buffer(_index, buffer):
“””
This function takes an index and a buffer.
The buffer is just an iterable of iterables (ex a list of lists)
Each buffer item is a row of values.
“””
filename = ‘run{}.csv’.format(_index)
with open(filename, ‘w’) as output:
writer = csv.writer(output)
writer.writerow([‘onset’, ‘type’]) # adding the heading
writer.writerows(buffer)

current_buffer = []
_index = 1

with open(‘localizer.log’, ‘rt’) as log_input:
for line in log_input:
# will deal ok with multi-space as long as
# you don’t care about the last column
onset, _type = line.split()[:2]
if not NEW_LOG_DELIMITER in line or not current_buffer:
# If it’s the first line (the current_buffer is empty)
# or the line does NOT contain “MYLOG” then
# collect it until it’s time to write it to file.
current_buffer.append([onset, _type])
else:
write_buffer(_index, current_buffer)
_index += 1
current_buffer = []
if current_buffer:
# We are now out of the loop,
# if there’s an unwritten buffer then write it to file.
write_buffer(_index, current_buffer)

Comments

comments