Hi kernel list,
I've got a few problems with 2.0.38.
All out of a sudden, our production web server crashes every other day.
It was running fine on 2.0.35 for a long time, but script kiddies forced the
upgrade to 2.0.38 and later to 2.0.39pre1 for the IP stack vulnerability to
be patched.
Even on that kernel, the system went smoothly for a couple of months, until
recently, when it started to crash frequently.
I was able to capture the ooopses as appended. Machine was rock solid and
needed a hard reboot though.
It is a Pentium 166, BusLogic SCSI, software RAID 1.
Kernel is 2.0.38 plus .39pre1 plus the OpenWall noexec-
stack (.38-ow4). The stack trace seem to point to the TCPIP layer, so
we might be talking about some new DoS vulnerability here, since the problems
started all out of the sudden.
Upgrading to 2.2 is planned, however cannot be done right now for other
reasons.
Any suggestions ?
Mario
----- Oops1-------
general protection: c634
CPU: 0
EIP: 0010:[<073f0813>]
EFLAGS: 00010202
eax: 073f0800 ebx: 02e69810 ecx: 05d75010 edx: 05d75000
esi: 00000434 edi: 00000000 ebp: 001dfb80 esp: 001d21a4
ds: 0018 es: 0018 fs: 002b gs: 0000 ss: 0018
Process swapper (pid: 0, process nr: 0, stackpage=001d0b00)
Stack: 0013c637 02e69810 02e69810 05d753b8 0013c937 02e69810 05d753b8 00003500
00000000 0000031c 0013cdc6 05d753b8 00000000 00003518 00000040 0019ea07
05d753b8 00000000 00000001 05d753b8 001dfc18 001dfb80 05d7502a 05d753b8
Call Trace: [<0013c637>] [<0013c937>] [<0013cdc6>] [<0019ea07>] [<0013d7cb>] [<0013d816>] [<00147505>]
[<0014f34e>] [<0014f376>] [<0014f8af>] [<0014f834>] [<00113419>] [<00118def>] [<0010ab2f>] [<001178e2>]
[<00117c44>] [<0010b0cf>] [<0010b4bc>] [<09000000>] [<08800000>] [<0010b6f2>] [<0010b4bc>] [<0010ad00>]
[<0013c637>] [<0013c937>] [<0013cdc6>] [<0019ea07>] [<0013d7cb>] [<0013d816>] [<00147505>] [<0014ef11>]
[<0014ca5c>] [<0014ccf5>] [<0014d741>] [<00144c43>] [<0013d9ec>] [<00118def>] [<0010ab2f>] [<001178e2>]
[<00117c44>] [<0010b0cf>] [<0010b4bc>] [<09000000>] [<08800000>] [<0010b6f2>] [<0010b4bc>] [<0010ad00>]
[<0013c637>] [<0013c937>] [<0014c44a>] [<0014d689>] [<00144c43>] [<0013d9ec>] [<00118def>] [<0010ab2f>]
[<001178df>] [<00117c44>] [<0010b0cf>] [<09000000>] [<08800000>] [<001b0018>] [<0011230e>] [<0011201c>]
[<00147505>] [<0010ad00>] [<0013c637>] [<0013c937>] [<0014c44a>] [<0014d689>] [<00144c43>] [<0013d9ec>]
[<00118def>] [<0010ab2f>] [<00109bc0>] [<0010ab9d>] [<001097bc>] [<00109578>]
Code: 07 10 98 e6 02 00 20 00 00 00 00 00 00 38 24 91 97 03 00 00
Aiee, killing interrupt handler
------ ksymoops for Ooops1 ----
Using `/usr/src/linux/System.map' to map addresses to symbols.
Trace: 13c637 <sock_wfree+23/2c>
Trace: 13c937 <kfree_skb+b7/f4>
Trace: 13cdc6 <dev_kfree_skb+3e/4c>
Trace: 19ea07 <ei_start_xmit+2eb/2f8>
Trace: 13d7cb <do_dev_queue_xmit+1c7/1f8>
Trace: 13d816 <dev_queue_xmit+1a/24>
Trace: 147505 <ip_queue_xmit+199/1ec>
Trace: 14f34e <tcp_write_wakeup+42a/440>
Trace: 14f376 <tcp_send_probe0+12/6c>
Trace: 14f8af <tcp_retransmit_timer+7b/e4>
Trace: 14f8af <tcp_retransmit_timer+7b/e4>
Trace: 113419 <timer_bh+2ed/334>
Trace: 118def <do_bottom_half+3b/60>
Trace: 10ab2f <handle_bottom_half+b/18>
Trace: 1178e2 <exit_notify+3e/1d8>
Trace: 117c44 <do_exit+1c8/1fc>
Trace: 10b0cf <die_if_kernel+2b7/2c0>
Trace: 10b4bc <do_general_protection>
Trace: 9000000
Trace: 8800000
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10ad00 <error_code+40/48>
Trace: 13c637 <sock_wfree+23/2c>
Trace: 13c937 <kfree_skb+b7/f4>
Trace: 13cdc6 <dev_kfree_skb+3e/4c>
Trace: 19ea07 <ei_start_xmit+2eb/2f8>
Trace: 13d7cb <do_dev_queue_xmit+1c7/1f8>
Trace: 13d816 <dev_queue_xmit+1a/24>
Trace: 147505 <ip_queue_xmit+199/1ec>
Trace: 14ef11 <tcp_send_ack+229/23c>
Trace: 14ca5c <tcp_queue+fc/184>
Trace: 14ccf5 <tcp_data+211/21c>
Trace: 14d741 <tcp_rcv+909/9b4>
Trace: 144c43 <ip_rcv+423/554>
Trace: 13d9ec <net_bh+fc/11c>
Trace: 118def <do_bottom_half+3b/60>
Trace: 10ab2f <handle_bottom_half+b/18>
Trace: 1178e2 <exit_notify+3e/1d8>
Trace: 117c44 <do_exit+1c8/1fc>
Trace: 10b0cf <die_if_kernel+2b7/2c0>
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 9000000
Trace: 8800000
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10ad00 <error_code+40/48>
Trace: 13c637 <sock_wfree+23/2c>
Trace: 13c937 <kfree_skb+b7/f4>
Trace: 14c44a <tcp_ack+57a/908>
Trace: 14d689 <tcp_rcv+851/9b4>
Trace: 144c43 <ip_rcv+423/554>
Trace: 13d9ec <net_bh+fc/11c>
Trace: 118def <do_bottom_half+3b/60>
Trace: 10ab2f <handle_bottom_half+b/18>
Trace: 1178df <exit_notify+3b/1d8>
Trace: 117c44 <do_exit+1c8/1fc>
Trace: 10b0cf <die_if_kernel+2b7/2c0>
Trace: 9000000
Trace: 8800000
Trace: 1b0018 <BusLogic_ProcDirectoryInfo+278/708>
Trace: 11230e <do_page_fault+2f2/304>
Trace: 11230e <do_page_fault+2f2/304>
Trace: 147505 <ip_queue_xmit+199/1ec>
Trace: 10ad00 <error_code+40/48>
Trace: 13c637 <sock_wfree+23/2c>
Trace: 13c937 <kfree_skb+b7/f4>
Trace: 14c44a <tcp_ack+57a/908>
Trace: 14d689 <tcp_rcv+851/9b4>
Trace: 144c43 <ip_rcv+423/554>
Trace: 13d9ec <net_bh+fc/11c>
Trace: 118def <do_bottom_half+3b/60>
Trace: 10ab2f <handle_bottom_half+b/18>
Trace: 109bc0 <sys_idle+5c/70>
Trace: 10ab9d <system_call+55/7c>
Trace: 1097bc <init>
Trace: 109578 <start_kernel+1d4/1e0>
Code:
Code: 07 popl %es
Code: 10 98 e6 02 00 adcb %bl,0x200002e6(%eax)
Code: 20
Code: 00 00 addb %al,(%eax)
Code: 00 00 addb %al,(%eax)
Code: 00 00 addb %al,(%eax)
Code: 38 24 91 cmpb %ah,(%ecx,%edx,4)
Code: 97 xchgl %eax,%edi
Code: 03 00 addl (%eax),%eax
Code: 00 00 addb %al,(%eax)
Code: 90 nop
Code: 90 nop
Code: 90 nop
---Ooops2 -----
general protection: c634
CPU: 0
EIP: 0010:[<073f0813>]
EFLAGS: 00010202
eax: 073f0800 ebx: 02e69810 ecx: 00000000 edx: 00000000
esi: 00000764 edi: 00000000 ebp: 001d2094 esp: 001d2028
ds: 0018 es: 0018 fs: 002b gs: 0000 ss: 0018
Process swapper (pid: 0, process nr: 0, stackpage=001d0b00)
Stack: 0013c637 02e69810 02e69810 066516e8 0013c937 02e69810 066516e8 066516e8
02e698c4 02e69810 00153e76 066516e8 00000000 02e69810 00000005 00000001
001440e6 02e69810 02e69810 00144024 00113419 02e69810 00000001 ffffffff
Call Trace: [<0013c637>] [<0013c937>] [<00153e76>] [<001440e6>] [<00144024>] [<00113419>] [<00118def>]
[<0010ab2f>] [<001178e2>] [<00117c44>] [<0010b0cf>] [<0010b4bc>] [<09000000>] [<08800000>] [<0010b6f2>]
[<0010b4bc>] [<0010ad00>] [<0013c637>] [<0013c937>] [<0013cdc6>] [<0019ea07>] [<0013d7cb>] [<0013d816>]
[<00147505>] [<0014f34e>] [<0014f376>] [<0014f8af>] [<0014f834>] [<00113419>] [<00118def>] [<0010ab2f>]
[<001178e2>] [<00117c44>] [<0010b0cf>] [<0010b4bc>] [<09000000>] [<08800000>] [<0010b6f2>] [<0010b4bc>]
[<0010ad00>] [<0013c637>] [<0013c937>] [<0013cdc6>] [<0019ea07>] [<0013d7cb>] [<0013d816>] [<00147505>]
[<0014ef11>] [<0014ca5c>] [<0014ccf5>] [<0014d741>] [<00144c43>] [<0013d9ec>] [<00118def>] [<0010ab2f>]
[<001178e2>] [<00117c44>] [<0010b0cf>] [<0010b4bc>] [<09000000>] [<08800000>] [<0010b6f2>] [<0010b4bc>]
[<0010ad00>] [<0013c637>] [<0013c937>] [<0014c44a>] [<0014d689>] [<00144c43>] [<0013d9ec>] [<00118def>]
[<0010ab2f>] [<001178df>] [<00117c44>] [<0010b0cf>] [<09000000>] [<08800000>] [<001b0018>] [<0011230e>]
[<0011201c>] [<00147505>] [<0010ad00>] [<0013c637>] [<0013c937>] [<0014c44a>] [<0014d689>] [<00144c43>]
[<0013d9ec>] [<00118def>] [<0010ab2f>] [<00109bc0>] [<0010ab9d>] [<001097bc>] [<00109578>]
Code: 07 10 98 e6 02 00 20 00 00 00 00 00 00 38 24 91 97 03 00 00
Aiee, killing interrupt handler
---- Ksymoops Ooops2 ----
Using `/usr/src/linux/System.map' to map addresses to symbols.
Trace: 13c637 <sock_wfree+23/2c>
Trace: 13c937 <kfree_skb+b7/f4>
Trace: 153e76 <destroy_sock+96/2cc>
Trace: 1440e6 <net_timer+c2/140>
Trace: 1440e6 <net_timer+c2/140>
Trace: 113419 <timer_bh+2ed/334>
Trace: 118def <do_bottom_half+3b/60>
Trace: 10ab2f <handle_bottom_half+b/18>
Trace: 1178e2 <exit_notify+3e/1d8>
Trace: 117c44 <do_exit+1c8/1fc>
Trace: 10b0cf <die_if_kernel+2b7/2c0>
Trace: 10b4bc <do_general_protection>
Trace: 9000000
Trace: 8800000
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10ad00 <error_code+40/48>
Trace: 13c637 <sock_wfree+23/2c>
Trace: 13c937 <kfree_skb+b7/f4>
Trace: 13cdc6 <dev_kfree_skb+3e/4c>
Trace: 19ea07 <ei_start_xmit+2eb/2f8>
Trace: 13d7cb <do_dev_queue_xmit+1c7/1f8>
Trace: 13d816 <dev_queue_xmit+1a/24>
Trace: 147505 <ip_queue_xmit+199/1ec>
Trace: 14f34e <tcp_write_wakeup+42a/440>
Trace: 14f376 <tcp_send_probe0+12/6c>
Trace: 14f8af <tcp_retransmit_timer+7b/e4>
Trace: 14f8af <tcp_retransmit_timer+7b/e4>
Trace: 113419 <timer_bh+2ed/334>
Trace: 118def <do_bottom_half+3b/60>
Trace: 10ab2f <handle_bottom_half+b/18>
Trace: 1178e2 <exit_notify+3e/1d8>
Trace: 117c44 <do_exit+1c8/1fc>
Trace: 10b0cf <die_if_kernel+2b7/2c0>
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 9000000
Trace: 8800000
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10ad00 <error_code+40/48>
Trace: 13c637 <sock_wfree+23/2c>
Trace: 13c937 <kfree_skb+b7/f4>
Trace: 13cdc6 <dev_kfree_skb+3e/4c>
Trace: 19ea07 <ei_start_xmit+2eb/2f8>
Trace: 13d7cb <do_dev_queue_xmit+1c7/1f8>
Trace: 13d816 <dev_queue_xmit+1a/24>
Trace: 147505 <ip_queue_xmit+199/1ec>
Trace: 14ef11 <tcp_send_ack+229/23c>
Trace: 14ca5c <tcp_queue+fc/184>
Trace: 14ccf5 <tcp_data+211/21c>
Trace: 14d741 <tcp_rcv+909/9b4>
Trace: 144c43 <ip_rcv+423/554>
Trace: 13d9ec <net_bh+fc/11c>
Trace: 118def <do_bottom_half+3b/60>
Trace: 10ab2f <handle_bottom_half+b/18>
Trace: 1178e2 <exit_notify+3e/1d8>
Trace: 117c44 <do_exit+1c8/1fc>
Trace: 10b0cf <die_if_kernel+2b7/2c0>
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 9000000
Trace: 8800000
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10b6f2 <do_general_protection+236/400>
Trace: 10ad00 <error_code+40/48>
Trace: 13c637 <sock_wfree+23/2c>
Trace: 13c937 <kfree_skb+b7/f4>
Trace: 14c44a <tcp_ack+57a/908>
Trace: 14d689 <tcp_rcv+851/9b4>
Trace: 144c43 <ip_rcv+423/554>
Trace: 13d9ec <net_bh+fc/11c>
Trace: 118def <do_bottom_half+3b/60>
Trace: 10ab2f <handle_bottom_half+b/18>
Trace: 1178df <exit_notify+3b/1d8>
Trace: 117c44 <do_exit+1c8/1fc>
Trace: 10b0cf <die_if_kernel+2b7/2c0>
Trace: 9000000
Trace: 8800000
Trace: 1b0018 <BusLogic_ProcDirectoryInfo+278/708>
Trace: 11230e <do_page_fault+2f2/304>
Trace: 11230e <do_page_fault+2f2/304>
Trace: 147505 <ip_queue_xmit+199/1ec>
Trace: 10ad00 <error_code+40/48>
Trace: 13c637 <sock_wfree+23/2c>
Trace: 13c937 <kfree_skb+b7/f4>
Trace: 14c44a <tcp_ack+57a/908>
Trace: 14d689 <tcp_rcv+851/9b4>
Trace: 144c43 <ip_rcv+423/554>
Trace: 13d9ec <net_bh+fc/11c>
Trace: 118def <do_bottom_half+3b/60>
Trace: 10ab2f <handle_bottom_half+b/18>
Trace: 109bc0 <sys_idle+5c/70>
Trace: 10ab9d <system_call+55/7c>
Trace: 1097bc <init>
Trace: 109578 <start_kernel+1d4/1e0>
Code:
Code: 07 popl %es
Code: 10 98 e6 02 00 adcb %bl,0x200002e6(%eax)
Code: 20
Code: 00 00 addb %al,(%eax)
Code: 00 00 addb %al,(%eax)
Code: 00 00 addb %al,(%eax)
Code: 38 24 91 cmpb %ah,(%ecx,%edx,4)
Code: 97 xchgl %eax,%edi
Code: 03 00 addl (%eax),%eax
Code: 00 00 addb %al,(%eax)
Code: 90 nop
Code: 90 nop
Code: 90 nop
--- Ooops 3, syslog capture (includes syms), httpd-s is apache 1.3.12
Unable to handle kernel paging request at virtual address fc0960a5
current->tss.cr3 = 04b5d000, %cr3 = 04b5d000
*pde = 00000000
Oops: 0002
CPU: 0
EIP: 0010:[<06f10c1b>]
EFLAGS: 00010212
eax: 06f10c0c ebx: 058bfc0c ecx: 04810010 edx: 04810000
esi: 00000124 edi: 00000000 ebp: 79c8a15d esp: 075f9d64
ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Process httpd-s (pid: 2125, process nr: 63, stackpage=075f9000)
Stack: 0013c6fb 058bfc0c 058bfc0c 048100a8 0013c9fb 058bfc0c 048100a8 048100a8
058bfc0c 00000000 0014c50e 048100a8 00000000 058bfc0c 017b1b20 4e096120
017b1b3c 017b1b20 00000000 ee073f00 001dfe01 00000001 000000ff 00000000
Call Trace: [sock_wfree+35/44] [kfree_skb+183/244] [tcp_ack+1402/2312] [tcp_rcv+2129/2484] [ip_rcv+1059/1364] [net_bh+252/284] [do_bottom_half+59/96]
[handle_bottom_half+11/24] [cleanup_rbuf+12/148] [tcp_recvmsg+1002/1036] [inet_recvmsg+114/136] [sock_read+171/192] [sys_read+192/232] [system_call+85/124]
Code: 00 64 c2 45 2c 03 00 00 00 c6 d1 9b 24 c6 d1 9b 24 c3 21 1d
-----Ooops 4 , syslog capture ------
Unable to handle kernel paging request at virtual address fc0960a5
current->tss.cr3 = 04b5d000, %cr3 = 04b5d000
*pde = 00000000
Oops: 0002
CPU: 0
EIP: 0010:[<06f10c1b>]
EFLAGS: 00010212
eax: 06f10c0c ebx: 058bfc0c ecx: 04810010 edx: 04810000
esi: 00000124 edi: 00000000 ebp: 79c8a15d esp: 075f9d64
ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Process httpd-s (pid: 2125, process nr: 63, stackpage=075f9000)
Stack: 0013c6fb 058bfc0c 058bfc0c 048100a8 0013c9fb 058bfc0c 048100a8 048100a8
058bfc0c 00000000 0014c50e 048100a8 00000000 058bfc0c 017b1b20 4e096120
017b1b3c 017b1b20 00000000 ee073f00 001dfe01 00000001 000000ff 00000000
Call Trace: [sock_wfree+35/44] [kfree_skb+183/244] [tcp_ack+1402/2312] [tcp_rcv+2129/2484] [ip_rcv+1059/1364] [net_bh+252/284] [do_bottom_half+59/96]
[handle_bottom_half+11/24] [cleanup_rbuf+12/148] [tcp_recvmsg+1002/1036] [inet_recvmsg+114/136] [sock_read+171/192] [sys_read+192/232] [system_call+85/124]
Code: 00 64 c2 45 2c 03 00 00 00 c6 d1 9b 24 c6 d1 9b 24 c3 21 1d
Aiee, killing interrupt handler
----- Ooops 5 ---------
Unable to handle kernel paging request at virtual address f9cc10a5
current->tss.cr3 = 05613000, %cr3 = 05613000
*pde = 00000000
Oops: 0002
CPU: 0
EIP: 0010:[<06f10c1b>]
EFLAGS: 00010216
eax: 06f10c0c ebx: 058bfc0c ecx: 0243b9fc edx: 0243b000
esi: 00000114 edi: 00000000 ebp: 001dfe4c esp: 05192e88
ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Process httpd-s (pid: 4783, process nr: 80, stackpage=05192000)
Stack: 0013c6fb 058bfc0c 058bfc0c 0243ba84 0013c9fb 058bfc0c 0243ba84 00003500
00000000 0000003c 0013ce8a 0243ba84 00000000 00003518 00000040 0019eccb
0243ba84 00000000 00000001 0243ba84 001dfee4 001dfe4c 0243ba16 0243ba84
Call Trace: [sock_wfree+35/44] [kfree_skb+183/244] [dev_kfree_skb+62/76] [ei_start_xmit+747/760] [do_dev_queue_xmit+455/504] [dev_queue_xmit+26/36] [ip_queue_xmit+409/492]
[tcp_send_ack+553/572] [tcp_delack_timer+0/16] [tcp_delack_timer+10/16] [timer_bh+749/820] [do_bottom_half+59/96] [handle_bottom_half+11/24] [sd_init+213/544]
Code: 00 64 c2 45 2c 03 00 00 00 c6 d1 9b 24 c6 d1 9b 24 c3 21 1d
Aiee, killing interrupt handler
------ Ooops 6 -------
general protection: 0000
CPU: 0
EIP: 0010:[def_callback3+15/60]
EFLAGS: 00010246
eax: 00154400 ebx: 0545a018 ecx: 0243b408 edx: 00000124
esi: 00000124 edi: 00000000 ebp: 18a9ff79 esp: 01946e80
ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Process mysqld (pid: 26867, process nr: 77, stackpage=01946000)
Stack: 0545a018 0013c6fb 0545a018 0545a018 0243b4a0 0013c9fb 0545a018 0243b4a0
0243b4a0 0545a018 00000000 0014c50e 0243b4a0 00000000 0545a018 0663d148
4e2f7720 0663d164 0663d148 00000000 00089d00 06b03c01 00080001 000000ff
Jul 19 00:04:01 multiweb kerneld: error: exit: Identifier removed
Call Trace: [sock_wfree+35/44] [kfree_skb+183/244] [tcp_ack+1402/2312] [tcp_rcv+2129/2484] [ip_rcv+1059/1364] [net_bh+252/284] [do_bottom_half+59/96]
[handle_bottom_half+11/24]
Code: 89 4c ff cf 39 83 3c 01 00 00 7c 1d 8b 83 d0 00 00 00 50 e8
Aiee, killing interrupt handler
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Mon Jul 24 2000 - 09:16:23 EDT