This is a comparison of the support various languages and tools have for reading and processing arbitrarily long lines of text. This gives an idea of the base overhead of using these tools as filters.

I originally ran these tests over 6 years ago, and it's interesting to compare those results now. All results have improved of course in line with the better hardware, but awk, python and tcl have made additional notable improvements over the versions available in 2002.

Each program loops reading a line from stdin and outputting immediately to stdout, and was tested/timed using:
time ./test_prog <test_data.txt >/dev/null
There were 2 test files each containing 354371 lines of text. The short lines (/usr/share/dict/words) average line length was 8 characters (excluding the LF) and long lines (/usr/share/doc/*) where the average line length was 37 chars. Many runs were done and the average taken so that no disk access was involved etc.
language/tool                           short lines (~8 chars)        long lines (~37 chars)
C (getline)                             0.082s  (0.072s/0.003s)       0.110s  (0.092s/0.012s)
C (linebuffer getc unlocked)            0.067s  (0.060s/0.001s)       0.138s  (0.117s/0.015s)
C (linebuffer fgets NoNulls unlocked)   0.109s  (0.097s/0.006s)       0.162s  (0.145s/0.011s)
C (linebuffer fgets unlocked)           0.160s  (0.144s/0.003s)       0.221s  (0.193s/0.011s)
C (linebuffer fgets NoNulls)            0.137s  (0.120s/0.001s)       0.189s  (0.163s/0.013s)
C (linebuffer fgets)                    0.178s  (0.161s/0.003s)       0.233s  (0.209s/0.012s)
C (linebuffer getc)                     0.067s  (0.057s/0.004s)       0.137s  (0.122s/0.009s)
grep (2.5.1)                            0.573s  (0.557s/0.003s)       1.496s  (1.469s/0.014s)
cgrep (2.5.1)                           0.109s  (0.098s/0.006s)       0.135s  (0.120s/0.010s)
sed (4.1.5)                             0.176s  (0.163s/0.006s)       0.195s  (0.176s/0.013s)
perl (5.8.8)                            0.371s  (0.220s/0.008s)       0.337s  (0.279s/0.011s)
awk (3.1.5)                             0.164s  (0.149s/0.003s)       0.195s  (0.174s/0.008s)
cpp                                     0.741s  (0.616s/0.119s)       2.145s  (1.842s/0.134s)
python (2.5.1)                          0.746s  (0.734s/0.006s)       0.825s  (0.803s/0.014s)
bash (3.2.25)                          12.769s (11.230s/1.193s)      27.956s (24.836s/2.280s)
dash (0.5.4)                            3.899s  (1.616s/2.275s)      14.891s  (4.569s/9.813s)
tcl (8.4.15)                            2.361s  (1.821s/0.176s)       2.375s  (2.005s/0.194s)
cat*                                    0.004s  (0.000s/0.004s)       0.014s  (0.001s/0.013s)

*cat actually does essentially the same since we're not actually changing the data :-)
however it does illustrate the advantages of using the optimum device block size and
bypassing stdio buffering etc.

You can download all source code used for the test.

Test host info:
        00:00.0 Host bridge: Intel Corporation Mobile 915GM/PM/GMS/910GML Express Processor to DRAM Controller (rev 03)

        vendor_id       : GenuineIntel
        cpu family      : 6
        model           : 13
        model name      : Intel(R) Pentium(R) M processor 1.70GHz
        stepping        : 8
        cpu MHz         : 1700.000
        cache size      : 2048 KB

        Linux #1 SMP Sat Apr 19 12:39:34 EDT 2008

        CFLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 \
                -m32 -march=i686 -mtune=generic -fasynchronous-unwind-tables"

        CFLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 \
                -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables"

© May 9 2008