|Source code (in)||csv.c|
|Source code (out)||csv.c|
|MIME type||text/csv, text/csv;header|
CSV is the abbreviation for Comma separated values. It is a text file format where data is arranged in a tabular representation. CSV files traditionally were used with spreadsheet calculation software, but has also been used as an import and export format for signal analysis software.
In the context of signal analysis, text rows typically relate to time (the number of sample sets taken, one text line per point in time), columns contain the values within a sample set (one value per column). There is a convention that column captions can be kept in a so-called header line.
The text files usually contain all-printable characters, and use CR/LF for text line termination. Some software versions also accept alternative line termination sequences, like LF only, or CR only.
Columns traditionally are separated by comma. But over time other separators were also used, and may optionally be supported by application software.
Numbers are communicated without specific marking (example: just "123.45" or "-12"). Floating point numbers may use exponential representation (example: "100.0E-3"). Text may or may not be enclosed in a pair of double quotes. There can be other data types for cells like dates, currency, etc, which shall not be discussed here.
Some applications support the notion of comments, which span from a comment leader (usually semi-colon) up to the end of the line. Applications may or may not accept "trailing" comments after data, or require that comments span a complete line on their own.
Since there is a great variety of software and platforms, it's best to generate output files conservatively, and accept the minimum feature set, and optionally a few variants which are rather popular in the field.
- The file size is not limited, neither is the line length, nor the acceptable set of used characters (specific applications may be limited, but the file format itself is not).
- Line termination sequences, column separators, and comment leaders have standard values (CR/LF, comma, semi-colon), but users may want to override these.
- There is no reliable condition to determine that a file contains CSV data. The .csv file extension is a convention but need not be used. Application software need not have access to "a filename" either. This weakens the support for automatic detection of the file format.
It's important to notice that the sigrok project implements support for using CSV formatted data as an input format as well as an output format. These are separate code paths, and need not be symmetric nor identical in their feature set or default behaviour. Given that the input path needs to accept text and translate it to the internal .sr presentation, and that the output path dumps the internally available .sr presentation to a text file, not all concepts translate equally well at either side. Aside from that conceptual difference, other factors like available time and manpower may influence the completeness or reliability of alternate code paths, not all are touched at the same moment by the same persons addressing the same requirements.
CSV input module
The input module for the CSV file format implements these features:
- Supports logic data, analog data, and timestamps.
- Logic data can use individual columns per data bit (the default), or can combine multiple bits in a column. For multi-bit columns the user can specify which number representation applies (binary, octal, hexadecimal) and how many bits the column contains.
- Analog values always occupy one column per value.
- The order of columns and their data types are up to the user, but must be specified when they differ from the default case. Even for the default layout it is a good idea to specify the format for the purpose of being explicit to avoid surprises, and to become self documenting.
- The default case is all-logic data with one bit per column in all columns of a text line ("multi-column mode") for backwards compatibility.
- "Simple" layouts can be described in a backwards compatible style with specific keywords, but the generic format specifier is most flexible and feature complete, and is considered to be as accessible to users and readable, so it should be preferred.
- For the single column case where a multi-bit value spans the corresponding number of logic channels, users need to specify the column number and channel count, and the number format if it is not binary.
- For multiple adjacent columns which each contain one bit of logic data (so-called "multi-column mode") the first column number must be specified. The channel count is optional and defaults to the remaining number of columns in the text line. The number format does not apply to single-bit data (binary fits as an implementation detail).
- A more generic "column formats" feature can express the above backwards compatible modes. But also allows the use of single- and multi-bit data in any number format and order of text columns, as well as analog input data (so-called "mixed signal" source), as well as optional timestamps which allow the automatic determination of the input data's samplerate for simple cases.
- A number of text lines at the top of the file can get skipped when necessary. The default is to process the complete file.
- An optional header line can be used to determine channel names (an exclusive feature to multi-column mode, off by default, simple channel names get assigned when the feature is off or the column lacks input text).
- The line termination gets auto-detected. The column separator and comment leader are user adjustable.
- Blank lines and comment-only lines get skipped (but are counted where line numbers get referenced).
- Users can specify a samplerate (by means of options, outside of the file). In the absence of a user spec but the presence of timestamps in the file the samplerate will get determined from input data.
- There are workarounds for odd platforms/applications which don't terminate the last text line in the file, or specify which one of one bytes goes first (useless use of BOM award).
- Automatic file format detection is supported, which enables the use of simple multi-column data files without header lines, without the need for user intervention or option specification.
- "Funnies" like double double quotes (escapes), embedded CRLF in fields enclosed in double quotes, etc. are not supported.
Notice that in 2019-12 user visible option names were renamed for consistency with other parts of the project, examples and existing external scripts need adjustment (interactive use in GUI applications should not notice).
List of options and builtin help text for the CSV input module:
$ sigrok-cli -I csv --show ID: csv Name: CSV Description: Comma-separated values Options: column_formats: Text columns data types. A comma separated list of [<cols>]<fmt>[<bits>] items. * for all remaining columns. - ignores columns, x/o/b/l logic data, a (and digits) analog data, t timestamps. (default ) single_column: Simple single-column mode, exclusively use text from the specified column (number starting at 1). Obsoleted by 'column_formats=4-,x16'. (default 0) first_column: First column with logic data in simple multi-column mode (number starting at 1, default 1). Obsoleted by 'column_formats=4-,*l'. (default 1) logic_channels: Logic channel count, required in simple single-column mode, defaults to "all remaining columns" in simple multi-column mode. Obsoleted by 'column_formats=8l'. (default 0) single_format: The input text number format of simple single-column mode: bin, hex, oct. Obsoleted by 'column_formats=x8'. (default 'bin', possible values 'bin', 'hex', 'oct') start_line: The line number at which to start processing input text (default: 1). (default 1) header: Use the first processed line's column captions (when available) as channel names. Off by default (default false) samplerate: The input data's sample rate in Hz. No default value. (default 0) column_separator: The sequence which separates text columns. Non-empty text, comma by default. (default ',') comment_leader: The text which starts comments at the end of text lines, semicolon by default. (default ';')
CSV output module
$ sigrok-cli -O csv --show ID: csv Name: CSV Description: Comma-separated values Options: gnuplot: gnuplot script file name (default ) scale: Scale gnuplot graphs (default true) value: Character to print between values (default ',') record: String to print between records (default '\n') frame: String to print between frames (default '\n') comment: String used at start of comment lines (default ';') header: Output header comment with capture metdata (default true) label: Type of column labels (default 'units', possible values 'units', 'channel', 'off') time: Output sample time as column 1 (default true) trigger: Output trigger indicator as last column (default false) dedup: Set to false to output duplicate rows (default false)
CSV input module
Simple multi-column data without header (the default format of the input module).
$ cat simple-multi-column-no-header.csv 1,0,1,0,1,1 0,1,0,1,0,1 1,0,1,0,1,1
$ sigrok-cli -i simple-multi-column-no-header.csv libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 6/6 channels 0:101 1:010 2:101 3:010 4:101 5:111
(also works when the input module is selected while no options get specified) $ sigrok-cli -I csv -i simple-multi-column-no-header.csv libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 6/6 channels 0:101 1:010 2:101 3:010 4:101 5:111
(being explicit this time, and using a more general format spec, the default format in that case) $ sigrok-cli -I csv:column_format='*l' -i simple-multi-column-no-header.csv libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 6/6 channels 0:101 1:010 2:101 3:010 4:101 5:111
Simple multi-column data with header (users need to specify options, which involves selecting an input module).
$ cat simple-multi-column-with-header.csv a,b,c,d,e,f 1,0,1,0,1,1 0,1,0,1,0,1 1,0,1,0,1,1
$ sigrok-cli -I csv:header=yes -i simple-multi-column-with-header.csv libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 6/6 channels a:101 b:010 c:101 d:010 e:101 f:111
(using the more general format spec) $ sigrok-cli -I csv:header=yes:column_formats="*l" -i simple-multi-column-with-header.csv libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 6/6 channels a:101 b:010 c:101 d:010 e:101 f:111
Simple single-column data with hex numbers (needs module and options specs):
$ cat simple-single-column-number-formats.csv x,y,8,z,1000 x,y,d,z,1101 x,y,7,z,0111 x,y,2,z,0010
$ sigrok-cli -I csv:single_column=3:single_format=hex:logic_channels=4 -i simple-single-column-number-formats.csv libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 4/4 channels 0:0110 1:0011 2:0110 3:1100
(using the more general format spec) $ sigrok-cli -I csv:column_formats=2-,x4 -i simple-single-column-number-formats.csv libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 4/4 channels 0:0110 1:0011 2:0110 3:1100
(also works with other number formats) $ sigrok-cli -I csv:column_formats=4-,b4 -i simple-single-column-number-formats.csv libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 4/4 channels 0:0110 1:0011 2:0110 3:1100
Mixed signal data in arbitrary order. Timestamps and automatic samplerate.
$ cat mixed-signal-data.csv time,ch1,ch2,logic,ch3,gray4,ch4,ignore,bits3 0.000,25.00,50.00,0,75.00,0,0.00,0,000 0.001,26.00,51.00,1,76.00,1,1.00,1,001 0.002,27.00,52.00,0,77.00,3,2.00,2,010 0.003,28.00,53.00,1,78.00,2,3.00,3,011 0.004,29.00,54.00,0,79.00,6,4.00,4,100 0.005,30.00,55.00,1,80.00,7,5.00,5,101 0.006,31.00,56.00,0,81.00,5,6.00,6,110 0.007,32.00,57.00,1,82.00,4,7.00,7,111 0.008,33.00,58.00,0,83.00,c,8.00,8,000 0.009,34.00,59.00,1,84.00,d,9.00,9,001
(grab the data, ignore the first column) $ sigrok-cli -I csv:header=yes:column_formats=-,2a,l,a,x4,a,-,b3 -i mixed-signal-data.csv | head -n 8 libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 8/12 channels ch1: 25.000 ch1: 26.000 ch1: 27.000 ch1: 28.000 ch1: 29.000 ch1: 30.000
(user specified samplerate) $ sigrok-cli -I csv:header=yes:column_formats=-,2a,l,a,x4,a,-,b3:samplerate=8000 -i mixed-signal-data.csv | head -n 8 META samplerate: 8000 libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 8/12 channels at 8 kHz ch1: 25.000 ch1: 26.000 ch1: 27.000 ch1: 28.000 ch1: 29.000
(automatic samplerate) $ sigrok-cli -I csv:header=yes:column_formats=t,2a,l,a,x4,a,-,b3 -i mixed-signal-data.csv META samplerate: 1000 libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 8/12 channels at 1 kHz ch1: 25.000 ch1: 26.000 ... ch1: 33.000 ch1: 34.000 ch2: 50.000 ch2: 51.000 ... ch2: 58.000 ch2: 59.000 ch3: 75.000 ch3: 76.000 ... ch3: 83.000 ch3: 84.000 ch4: 0.000 ch4: 1.000 ... ch4: 8.000 ch4: 9.000 logic:01010101 01 gray4:01100110 01 gray4:00111100 00 gray4:00001111 11 gray4:00000000 11 bits3:01010101 01 bits3:00110011 00 bits3:00001111 00
Comments, empty lines, skipped lines.
$ cat -n comments-empty-skipped.csv 1 These lines neither are comments 2 nor are they header nor data lines. 3 It's some introductory text, captions, 4 or whatever -- let's not process that. 5 6 ; comments get trimmed and skipped out of the box 7 ; as are empty lines like above and below 8 9 ; yet another comment 10 1,0,1,0 11 0,1,0,1 12 1,0,1,0
$ sigrok-cli -I csv:start_line=5 -i comments-empty-skipped.csv libsigrok 0.6.0-git-b99c8ecdec45 Acquisition with 4/4 channels 0:101 1:010 2:101 3:010
CSV output module