Donald Baud via freebsd-net
2016-07-10 06:19:48 UTC
Hi I'm running an l2tp lns through mpd3.8 and it's been crashing twice in 24h.
This is a new project replacing a cisco 7206, 700-sessions 800mbit/s
I am not familiar with troubleshooting kernel panic's,
I am suspecting that the crash is happening inside the netgraph module because the crash is happening at the
instruction pointer = 0x20:0xffffffff81c38283
I included the 2 two crash logs. I need some help to to figure out what to do next.
-Dbaud
============================
The box is a:
# uname -a
FreeBSD mybox.example.com 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016 ***@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
# kldstat
Id Refs Address Size Name
1 34 0xffffffff80200000 17bc6a8 kernel
2 2 0xffffffff81c11000 114db ipfw.ko
3 1 0xffffffff81c23000 d32f dummynet.ko
4 1 0xffffffff81c31000 3831 ng_socket.ko
5 9 0xffffffff81c35000 ba02 netgraph.ko
6 1 0xffffffff81c41000 2b99 ng_mppc.ko
7 1 0xffffffff81c44000 80c rc4.ko
8 1 0xffffffff81c45000 23dc vmmemctl.ko
9 1 0xffffffff81c48000 397d ng_l2tp.ko
10 1 0xffffffff81c4c000 4b04 ng_ksocket.ko
11 1 0xffffffff81c51000 17d6 ng_tee.ko
12 1 0xffffffff81c53000 40d2 ng_iface.ko
13 1 0xffffffff81c58000 5829 ng_ppp.ko
14 1 0xffffffff81c5e000 18b1 ng_tcpmss.ko
15 1 0xffffffff81c60000 2df7 ng_vjc.ko
=======================================
First crash dump:
Jul 8 08:09:04 mybox syslogd: kernel boot file is /boot/kernel/kernel
Jul 8 08:09:04 mybox kernel:
Jul 8 08:09:04 mybox kernel:
Jul 8 08:09:04 mybox kernel: Fatal trap 12: page fault while in kernel mode
Jul 8 08:09:04 mybox kernel: cpuid = 1; apic id = 01
Jul 8 08:09:04 mybox kernel: fault virtual address = 0x28
Jul 8 08:09:04 mybox kernel: fault code = supervisor read data, page not present
Jul 8 08:09:04 mybox kernel: instruction pointer = 0x20:0xffffffff81c38283
Jul 8 08:09:04 mybox kernel: stack pointer = 0x28:0xfffffe0174d85540
Jul 8 08:09:04 mybox kernel: frame pointer = 0x28:0xfffffe0174d85580
Jul 8 08:09:04 mybox kernel: code segment = base 0x0, limit 0xfffff, type 0x1b
Jul 8 08:09:04 mybox kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Jul 8 08:09:04 mybox kernel: processor eflags = interrupt enabled, resume, IOPL = 0
Jul 8 08:09:04 mybox kernel: current process = 628 (ng_queue2)
Jul 8 08:09:04 mybox kernel: trap number = 12
Jul 8 08:09:04 mybox kernel: panic: page fault
Jul 8 08:09:04 mybox kernel: cpuid = 1
Jul 8 08:09:04 mybox kernel: KDB: stack backtrace:
Jul 8 08:09:04 mybox kernel: #0 0xffffffff8098e390 at kdb_backtrace+0x60
Jul 8 08:09:04 mybox kernel: #1 0xffffffff80951066 at vpanic+0x126
Jul 8 08:09:04 mybox kernel: #2 0xffffffff80950f33 at panic+0x43
Jul 8 08:09:04 mybox kernel: #3 0xffffffff80d55f7b at trap_fatal+0x36b
Jul 8 08:09:04 mybox kernel: #4 0xffffffff80d5627d at trap_pfault+0x2ed
Jul 8 08:09:04 mybox kernel: #5 0xffffffff80d558fa at trap+0x47a
Jul 8 08:09:04 mybox kernel: #6 0xffffffff80d3b8d2 at calltrap+0x8
Jul 8 08:09:04 mybox kernel: #7 0xffffffff81c5e509 at ng_tcpmss_rcvdata+0x2d9
Jul 8 08:09:04 mybox kernel: #8 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 08:09:04 mybox kernel: #9 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 08:09:04 mybox kernel: #10 0xffffffff81c5a1c8 at ng_ppp_comp_recv+0x148
Jul 8 08:09:04 mybox kernel: #11 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 08:09:04 mybox kernel: #12 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 08:09:04 mybox kernel: #13 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 08:09:04 mybox kernel: #14 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 08:09:04 mybox kernel: #15 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 08:09:04 mybox kernel: #16 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 08:09:04 mybox kernel: #17 0xffffffff81c4d3e2 at ng_ksocket_incoming2+0x2f2
Jul 8 08:09:04 mybox kernel: Uptime: 5d17h47m38s
Jul 8 08:09:04 mybox kernel: Copyright (c) 1992-2016 The FreeBSD Project.
Jul 8 08:09:04 mybox kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jul 8 08:09:04 mybox kernel: The Regents of the University of California. All rights reserved.
Jul 8 08:09:04 mybox kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jul 8 08:09:04 mybox kernel: FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016
Jul 8 08:09:04 mybox kernel: ***@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
Jul 8 08:09:04 mybox kernel: FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Jul 8 08:09:04 mybox kernel: CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.00-MHz K8-class CPU)
Jul 8 08:09:04 mybox kernel: Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7
Jul 8 08:09:04 mybox kernel: Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
Jul 8 08:09:04 mybox kernel: Features2=0x9fba2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
Jul 8 08:09:04 mybox kernel: AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Jul 8 08:09:04 mybox kernel: AMD Features2=0x1<LAHF>
Jul 8 08:09:04 mybox kernel: Structured Extended Features=0x2<TSCADJ>
Jul 8 08:09:04 mybox kernel: TSC: P-state invariant
Jul 8 08:09:04 mybox kernel: Hypervisor: Origin = "VMwareVMware"
Jul 8 08:09:04 mybox kernel: real memory = 6442450944 (6144 MB)
Jul 8 08:09:04 mybox kernel: avail memory = 6183518208 (5897 MB)
Jul 8 08:09:04 mybox kernel: Event timer "LAPIC" quality 600
Jul 8 08:09:04 mybox kernel: ACPI APIC Table: <PTLTD APIC >
Jul 8 08:09:04 mybox kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Jul 8 08:09:04 mybox kernel: FreeBSD/SMP: 1 package(s) x 4 core(s)
Jul 8 08:09:04 mybox kernel: cpu0 (BSP): APIC ID: 0
Jul 8 08:09:04 mybox kernel: cpu1 (AP): APIC ID: 1
Jul 8 08:09:04 mybox kernel: cpu2 (AP): APIC ID: 2
Jul 8 08:09:04 mybox kernel: cpu3 (AP): APIC ID: 3
Jul 8 08:09:04 mybox kernel: random: <Software, Yarrow> initialized
Jul 8 08:09:04 mybox kernel: MADT: Forcing active-low polarity and level trigger for SCI
Jul 8 08:09:04 mybox kernel: ioapic0 <Version 1.1> irqs 0-23 on motherboard
Jul 8 08:09:04 mybox kernel: kbd1 at kbdmux0
Jul 8 08:09:04 mybox kernel: acpi0: <INTEL 440BX> on motherboard
Jul 8 08:09:04 mybox kernel: acpi0: Power Button (fixed)
Jul 8 08:09:04 mybox kernel: Timecounter "HPET" frequency 14318180 Hz quality 950
Jul 8 08:09:04 mybox kernel: cpu0: <ACPI CPU> on acpi0
Jul 8 08:09:04 mybox kernel: cpu1: <ACPI CPU> on acpi0
Jul 8 08:09:04 mybox kernel: cpu2: <ACPI CPU> on acpi0
Jul 8 08:09:04 mybox kernel: cpu3: <ACPI CPU> on acpi0
Jul 8 08:09:04 mybox kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Jul 8 08:09:04 mybox kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Jul 8 08:09:04 mybox kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Jul 8 08:09:04 mybox kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Jul 8 08:09:04 mybox kernel: Event timer "RTC" frequency 32768 Hz quality 0
Jul 8 08:09:04 mybox kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Jul 8 08:09:04 mybox kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
Jul 8 08:09:04 mybox kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Jul 8 08:09:04 mybox kernel: pci0: <ACPI PCI bus> on pcib0
Jul 8 08:09:04 mybox kernel: pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
Jul 8 08:09:04 mybox kernel: pci1: <ACPI PCI bus> on pcib1
Jul 8 08:09:04 mybox kernel: isab0: <PCI-ISA bridge> at device 7.0 on pci0
Jul 8 08:09:04 mybox kernel: isa0: <ISA bus> on isab0
Jul 8 08:09:04 mybox kernel: atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
Jul 8 08:09:04 mybox kernel: ata0: <ATA channel> at channel 0 on atapci0
Jul 8 08:09:04 mybox kernel: ata1: <ATA channel> at channel 1 on atapci0
Jul 8 08:09:04 mybox kernel: pci0: <bridge> at device 7.3 (no driver attached)
Jul 8 08:09:04 mybox kernel: vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
Jul 8 08:09:04 mybox kernel: vgapci0: Boot video device
Jul 8 08:09:04 mybox kernel: mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0
Jul 8 08:09:04 mybox kernel: mpt0: MPI Version=1.2.0.0
Jul 8 08:09:04 mybox kernel: pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
Jul 8 08:09:04 mybox kernel: pci2: <ACPI PCI bus> on pcib2
Jul 8 08:09:04 mybox kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x2000-0x203f mem 0xfd5c0000-0xfd5dffff,0xfdff0000-0xfdffffff irq 18 at device 0.0 on pci2
Jul 8 08:09:04 mybox kernel: em0: Ethernet address: 00:50:56:8d:d7:57
Jul 8 08:09:04 mybox kernel: pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
Jul 8 08:09:04 mybox kernel: pci3: <ACPI PCI bus> on pcib3
Jul 8 08:09:04 mybox kernel: vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd4fc000-0xfd4fcfff,0xfd4fd000-0xfd4fdfff,0xfd4fe000-0xfd4fffff irq 18 at device 0.0 on pci3
Jul 8 08:09:04 mybox kernel: vmx0: Ethernet address: 00:50:56:8d:4c:68
Jul 8 08:09:04 mybox kernel: pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
Jul 8 08:09:04 mybox kernel: pci4: <ACPI PCI bus> on pcib4
Jul 8 08:09:04 mybox kernel: pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
Jul 8 08:09:04 mybox kernel: pci5: <ACPI PCI bus> on pcib5
Jul 8 08:09:04 mybox kernel: pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
Jul 8 08:09:04 mybox kernel: pci6: <ACPI PCI bus> on pcib6
Jul 8 08:09:04 mybox kernel: pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
Jul 8 08:09:04 mybox kernel: pci7: <ACPI PCI bus> on pcib7
Jul 8 08:09:04 mybox kernel: pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
Jul 8 08:09:04 mybox kernel: pci8: <ACPI PCI bus> on pcib8
Jul 8 08:09:04 mybox kernel: pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
Jul 8 08:09:04 mybox kernel: pci9: <ACPI PCI bus> on pcib9
Jul 8 08:09:04 mybox kernel: pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
Jul 8 08:09:04 mybox kernel: pci10: <ACPI PCI bus> on pcib10
Jul 8 08:09:04 mybox kernel: pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
Jul 8 08:09:04 mybox kernel: pci11: <ACPI PCI bus> on pcib11
Jul 8 08:09:04 mybox kernel: vmx1: <VMware VMXNET3 Ethernet Adapter> port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci11
Jul 8 08:09:04 mybox kernel: vmx1: Ethernet address: 00:50:56:8d:85:29
Jul 8 08:09:04 mybox kernel: pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
Jul 8 08:09:04 mybox kernel: pci12: <ACPI PCI bus> on pcib12
Jul 8 08:09:04 mybox kernel: pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
Jul 8 08:09:04 mybox kernel: pci13: <ACPI PCI bus> on pcib13
Jul 8 08:09:04 mybox kernel: pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
Jul 8 08:09:04 mybox kernel: pci14: <ACPI PCI bus> on pcib14
Jul 8 08:09:04 mybox kernel: pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
Jul 8 08:09:04 mybox kernel: pci15: <ACPI PCI bus> on pcib15
Jul 8 08:09:04 mybox kernel: pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
Jul 8 08:09:04 mybox kernel: pci16: <ACPI PCI bus> on pcib16
Jul 8 08:09:04 mybox kernel: pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
Jul 8 08:09:04 mybox kernel: pci17: <ACPI PCI bus> on pcib17
Jul 8 08:09:04 mybox kernel: pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
Jul 8 08:09:04 mybox kernel: pci18: <ACPI PCI bus> on pcib18
Jul 8 08:09:04 mybox kernel: pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
Jul 8 08:09:04 mybox kernel: pci19: <ACPI PCI bus> on pcib19
Jul 8 08:09:04 mybox kernel: pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
Jul 8 08:09:04 mybox kernel: pci20: <ACPI PCI bus> on pcib20
Jul 8 08:09:04 mybox kernel: pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
Jul 8 08:09:04 mybox kernel: pci21: <ACPI PCI bus> on pcib21
Jul 8 08:09:04 mybox kernel: pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
Jul 8 08:09:04 mybox kernel: pci22: <ACPI PCI bus> on pcib22
Jul 8 08:09:04 mybox kernel: pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
Jul 8 08:09:04 mybox kernel: pci23: <ACPI PCI bus> on pcib23
Jul 8 08:09:04 mybox kernel: pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
Jul 8 08:09:04 mybox kernel: pci24: <ACPI PCI bus> on pcib24
Jul 8 08:09:04 mybox kernel: pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
Jul 8 08:09:04 mybox kernel: pci25: <ACPI PCI bus> on pcib25
Jul 8 08:09:04 mybox kernel: pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
Jul 8 08:09:04 mybox kernel: pci26: <ACPI PCI bus> on pcib26
Jul 8 08:09:04 mybox kernel: pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
Jul 8 08:09:04 mybox kernel: pci27: <ACPI PCI bus> on pcib27
Jul 8 08:09:04 mybox kernel: pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
Jul 8 08:09:04 mybox kernel: pci28: <ACPI PCI bus> on pcib28
Jul 8 08:09:04 mybox kernel: pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
Jul 8 08:09:04 mybox kernel: pci29: <ACPI PCI bus> on pcib29
Jul 8 08:09:04 mybox kernel: pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
Jul 8 08:09:04 mybox kernel: pci30: <ACPI PCI bus> on pcib30
Jul 8 08:09:04 mybox kernel: pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
Jul 8 08:09:04 mybox kernel: pci31: <ACPI PCI bus> on pcib31
Jul 8 08:09:04 mybox kernel: pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
Jul 8 08:09:04 mybox kernel: pci32: <ACPI PCI bus> on pcib32
Jul 8 08:09:04 mybox kernel: pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
Jul 8 08:09:04 mybox kernel: pci33: <ACPI PCI bus> on pcib33
Jul 8 08:09:04 mybox kernel: pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
Jul 8 08:09:04 mybox kernel: pci34: <ACPI PCI bus> on pcib34
Jul 8 08:09:04 mybox kernel: acpi_acad0: <AC Adapter> on acpi0
Jul 8 08:09:04 mybox kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Jul 8 08:09:04 mybox kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Jul 8 08:09:04 mybox kernel: kbd0 at atkbd0
Jul 8 08:09:04 mybox kernel: atkbd0: [GIANT-LOCKED]
Jul 8 08:09:04 mybox kernel: psm0: <PS/2 Mouse> irq 12 on atkbdc0
Jul 8 08:09:04 mybox kernel: psm0: [GIANT-LOCKED]
Jul 8 08:09:04 mybox kernel: psm0: model IntelliMouse, device ID 3
Jul 8 08:09:04 mybox kernel: fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Jul 8 08:09:04 mybox kernel: fd0: <1440-KB 3.5" drive> on fdc0 drive 0
Jul 8 08:09:04 mybox kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
Jul 8 08:09:04 mybox kernel: sc0: <System console> at flags 0x100 on isa0
Jul 8 08:09:04 mybox kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Jul 8 08:09:04 mybox kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Jul 8 08:09:04 mybox kernel: ppc0: cannot reserve I/O port range
Jul 8 08:09:04 mybox kernel: Timecounters tick every 1.000 msec
Jul 8 08:09:04 mybox kernel: em0: link state changed to UP
Jul 8 08:09:04 mybox kernel: random: unblocking device.
Jul 8 08:09:04 mybox kernel: cd0 at ata1 bus 0 scbus1 target 0 lun 0
Jul 8 08:09:04 mybox kernel: cd0: <NECVMWar VMware IDE CDR10 1.00> Removable CD-ROM SCSI device
Jul 8 08:09:04 mybox kernel: cd0: Serial Number 10000000000000000001
Jul 8 08:09:04 mybox kernel: cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
Jul 8 08:09:04 mybox kernel: cd0: Attempt to query device size failed: NOT READY, Medium not present
Jul 8 08:09:04 mybox kernel: cd0: quirks=0x40<RETRY_BUSY>
Jul 8 08:09:04 mybox kernel: da0 at mpt0 bus 0 scbus2 target 0 lun 0
Jul 8 08:09:04 mybox kernel: da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
Jul 8 08:09:04 mybox kernel: da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
Jul 8 08:09:04 mybox kernel: da0: Command Queueing enabled
Jul 8 08:09:04 mybox kernel: da0: 12288MB (25165824 512 byte sectors)
Jul 8 08:09:04 mybox kernel: da0: quirks=0x40<RETRY_BUSY>
Jul 8 08:09:04 mybox kernel: SMP: AP CPU #3 Launched!
Jul 8 08:09:04 mybox kernel: SMP: AP CPU #1 Launched!
Jul 8 08:09:04 mybox kernel: SMP: AP CPU #2 Launched!
Jul 8 08:09:04 mybox kernel: Trying to mount root from ufs:/dev/da0p2 [rw]...
Jul 8 08:09:04 mybox kernel: WARNING: / was not properly dismounted
Jul 8 08:09:04 mybox kernel: vmx0: link state changed to UP
Jul 8 08:09:04 mybox kernel: vmx1: link state changed to UP
Jul 8 08:09:04 mybox kernel: Limiting icmp unreach response from 1816 to 200 packets/sec
Jul 8 08:09:04 mybox kernel: Limiting icmp unreach response from 1982 to 200 packets/sec
Jul 8 08:09:04 mybox kernel: ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to deny, logging disabled
Jul 8 08:09:04 mybox kernel: DUMMYNET 0 with IPv6 initialized (100409)
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched FIFO loaded
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched QFQ loaded
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched RR loaded
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched WF2Q+ loaded
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched PRIO loaded
Jul 8 08:09:04 mybox kernel: Limiting icmp unreach response from 2125 to 200 packets/sec
Jul 8 08:09:05 mybox kernel: WARNING: attempt to domain_add(netgraph) after domainfinalize()
Jul 8 08:09:06 mybox kernel: done
Jul 8 08:09:07 mybox kernel: VMware memory control driver initialized
Jul 8 08:09:07 mybox kernel: done
===========================================
Second crash dump
Jul 8 20:02:30 mybox syslogd: kernel boot file is /boot/kernel/kernel
Jul 8 20:02:30 mybox kernel:
Jul 8 20:02:30 mybox kernel:
Jul 8 20:02:30 mybox kernel: Fatal trap 12: page fault while in kernel mode
Jul 8 20:02:30 mybox kernel: cpuid = 2; apic id = 02
Jul 8 20:02:30 mybox kernel: fault virtual address = 0x28
Jul 8 20:02:30 mybox kernel: fault code = supervisor read data, page not present
Jul 8 20:02:30 mybox kernel: instruction pointer = 0x20:0xffffffff81c38283
Jul 8 20:02:30 mybox kernel: stack pointer = 0x28:0xfffffe0174d99660
Jul 8 20:02:30 mybox kernel: frame pointer = 0x28:0xfffffe0174d996a0
Jul 8 20:02:30 mybox kernel: code segment = base 0x0, limit 0xfffff, type 0x1b
Jul 8 20:02:30 mybox kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Jul 8 20:02:30 mybox kernel: processor eflags = interrupt enabled, resume, IOPL = 0
Jul 8 20:02:30 mybox kernel: current process = 627 (ng_queue3)
Jul 8 20:02:30 mybox kernel: trap number = 12
Jul 8 20:02:30 mybox kernel: panic: page fault
Jul 8 20:02:30 mybox kernel: cpuid = 2
Jul 8 20:02:30 mybox kernel: KDB: stack backtrace:
Jul 8 20:02:30 mybox kernel: #0 0xffffffff8098e390 at kdb_backtrace+0x60
Jul 8 20:02:30 mybox kernel: #1 0xffffffff80951066 at vpanic+0x126
Jul 8 20:02:30 mybox kernel: #2 0xffffffff80950f33 at panic+0x43
Jul 8 20:02:30 mybox kernel: #3 0xffffffff80d55f7b at trap_fatal+0x36b
Jul 8 20:02:30 mybox kernel: #4 0xffffffff80d5627d at trap_pfault+0x2ed
Jul 8 20:02:30 mybox kernel: #5 0xffffffff80d558fa at trap+0x47a
Jul 8 20:02:30 mybox kernel: #6 0xffffffff80d3b8d2 at calltrap+0x8
Jul 8 20:02:30 mybox kernel: #7 0xffffffff81c5a042 at ng_ppp_bypass+0x62
Jul 8 20:02:30 mybox kernel: #8 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 20:02:30 mybox kernel: #9 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 20:02:30 mybox kernel: #10 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 20:02:30 mybox kernel: #11 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 20:02:30 mybox kernel: #12 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 20:02:30 mybox kernel: #13 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 20:02:30 mybox kernel: #14 0xffffffff81c4d3e2 at ng_ksocket_incoming2+0x2f2
Jul 8 20:02:30 mybox kernel: #15 0xffffffff81c36f62 at ng_apply_item+0xb2
Jul 8 20:02:30 mybox kernel: #16 0xffffffff81c38d39 at ngthread+0x1b9
Jul 8 20:02:30 mybox kernel: #17 0xffffffff8091a4ea at fork_exit+0x9a
Jul 8 20:02:30 mybox kernel: Uptime: 11h52m56s
Jul 8 20:02:30 mybox kernel: Dumping 591 out of 6119 MB:..3%..11%..22%..33%..41%..52%..63%..71%..82%..92%
Jul 8 20:02:30 mybox kernel: Dump complete
Jul 8 20:02:30 mybox kernel: Copyright (c) 1992-2016 The FreeBSD Project.
Jul 8 20:02:30 mybox kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jul 8 20:02:30 mybox kernel: The Regents of the University of California. All rights reserved.
Jul 8 20:02:30 mybox kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jul 8 20:02:30 mybox kernel: FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016
Jul 8 20:02:30 mybox kernel: ***@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
Jul 8 20:02:30 mybox kernel: FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Jul 8 20:02:30 mybox kernel: CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.00-MHz K8-class CPU)
Jul 8 20:02:30 mybox kernel: Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7
Jul 8 20:02:30 mybox kernel: Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
Jul 8 20:02:30 mybox kernel: Features2=0x9fba2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
Jul 8 20:02:30 mybox kernel: AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Jul 8 20:02:30 mybox kernel: AMD Features2=0x1<LAHF>
Jul 8 20:02:30 mybox kernel: Structured Extended Features=0x2<TSCADJ>
Jul 8 20:02:30 mybox kernel: TSC: P-state invariant
Jul 8 20:02:30 mybox kernel: Hypervisor: Origin = "VMwareVMware"
Jul 8 20:02:30 mybox kernel: real memory = 6442450944 (6144 MB)
Jul 8 20:02:30 mybox kernel: avail memory = 6183518208 (5897 MB)
Jul 8 20:02:30 mybox kernel: Event timer "LAPIC" quality 600
Jul 8 20:02:30 mybox kernel: ACPI APIC Table: <PTLTD APIC >
Jul 8 20:02:30 mybox kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Jul 8 20:02:30 mybox kernel: FreeBSD/SMP: 1 package(s) x 4 core(s)
Jul 8 20:02:30 mybox kernel: cpu0 (BSP): APIC ID: 0
Jul 8 20:02:30 mybox kernel: cpu1 (AP): APIC ID: 1
Jul 8 20:02:30 mybox kernel: cpu2 (AP): APIC ID: 2
Jul 8 20:02:30 mybox kernel: cpu3 (AP): APIC ID: 3
Jul 8 20:02:30 mybox kernel: random: <Software, Yarrow> initialized
Jul 8 20:02:30 mybox kernel: MADT: Forcing active-low polarity and level trigger for SCI
Jul 8 20:02:30 mybox kernel: ioapic0 <Version 1.1> irqs 0-23 on motherboard
Jul 8 20:02:30 mybox kernel: kbd1 at kbdmux0
Jul 8 20:02:30 mybox kernel: acpi0: <INTEL 440BX> on motherboard
Jul 8 20:02:30 mybox kernel: acpi0: Power Button (fixed)
Jul 8 20:02:30 mybox kernel: Timecounter "HPET" frequency 14318180 Hz quality 950
Jul 8 20:02:30 mybox kernel: cpu0: <ACPI CPU> on acpi0
Jul 8 20:02:30 mybox kernel: cpu1: <ACPI CPU> on acpi0
Jul 8 20:02:30 mybox kernel: cpu2: <ACPI CPU> on acpi0
Jul 8 20:02:30 mybox kernel: cpu3: <ACPI CPU> on acpi0
Jul 8 20:02:30 mybox kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Jul 8 20:02:30 mybox kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Jul 8 20:02:30 mybox kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Jul 8 20:02:30 mybox kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Jul 8 20:02:30 mybox kernel: Event timer "RTC" frequency 32768 Hz quality 0
Jul 8 20:02:30 mybox kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Jul 8 20:02:30 mybox kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
Jul 8 20:02:30 mybox kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Jul 8 20:02:30 mybox kernel: pci0: <ACPI PCI bus> on pcib0
Jul 8 20:02:30 mybox kernel: pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
Jul 8 20:02:30 mybox kernel: pci1: <ACPI PCI bus> on pcib1
Jul 8 20:02:30 mybox kernel: isab0: <PCI-ISA bridge> at device 7.0 on pci0
Jul 8 20:02:30 mybox kernel: isa0: <ISA bus> on isab0
Jul 8 20:02:30 mybox kernel: atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
Jul 8 20:02:30 mybox kernel: ata0: <ATA channel> at channel 0 on atapci0
Jul 8 20:02:30 mybox kernel: ata1: <ATA channel> at channel 1 on atapci0
Jul 8 20:02:30 mybox kernel: pci0: <bridge> at device 7.3 (no driver attached)
Jul 8 20:02:30 mybox kernel: vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
Jul 8 20:02:30 mybox kernel: vgapci0: Boot video device
Jul 8 20:02:30 mybox kernel: mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0
Jul 8 20:02:30 mybox kernel: mpt0: MPI Version=1.2.0.0
Jul 8 20:02:30 mybox kernel: pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
Jul 8 20:02:30 mybox kernel: pci2: <ACPI PCI bus> on pcib2
Jul 8 20:02:30 mybox kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x2000-0x203f mem 0xfd5c0000-0xfd5dffff,0xfdff0000-0xfdffffff irq 18 at device 0.0 on pci2
Jul 8 20:02:30 mybox kernel: em0: Ethernet address: 00:50:56:8d:d7:57
Jul 8 20:02:30 mybox kernel: pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
Jul 8 20:02:30 mybox kernel: pci3: <ACPI PCI bus> on pcib3
Jul 8 20:02:30 mybox kernel: vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd4fc000-0xfd4fcfff,0xfd4fd000-0xfd4fdfff,0xfd4fe000-0xfd4fffff irq 18 at device 0.0 on pci3
Jul 8 20:02:30 mybox kernel: vmx0: Ethernet address: 00:50:56:8d:4c:68
Jul 8 20:02:30 mybox kernel: pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
Jul 8 20:02:30 mybox kernel: pci4: <ACPI PCI bus> on pcib4
Jul 8 20:02:30 mybox kernel: pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
Jul 8 20:02:30 mybox kernel: pci5: <ACPI PCI bus> on pcib5
Jul 8 20:02:30 mybox kernel: pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
Jul 8 20:02:30 mybox kernel: pci6: <ACPI PCI bus> on pcib6
Jul 8 20:02:30 mybox kernel: pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
Jul 8 20:02:30 mybox kernel: pci7: <ACPI PCI bus> on pcib7
Jul 8 20:02:30 mybox kernel: pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
Jul 8 20:02:30 mybox kernel: pci8: <ACPI PCI bus> on pcib8
Jul 8 20:02:30 mybox kernel: pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
Jul 8 20:02:30 mybox kernel: pci9: <ACPI PCI bus> on pcib9
Jul 8 20:02:30 mybox kernel: pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
Jul 8 20:02:30 mybox kernel: pci10: <ACPI PCI bus> on pcib10
Jul 8 20:02:30 mybox kernel: pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
Jul 8 20:02:30 mybox kernel: pci11: <ACPI PCI bus> on pcib11
Jul 8 20:02:30 mybox kernel: vmx1: <VMware VMXNET3 Ethernet Adapter> port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci11
Jul 8 20:02:30 mybox kernel: vmx1: Ethernet address: 00:50:56:8d:85:29
Jul 8 20:02:30 mybox kernel: pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
Jul 8 20:02:30 mybox kernel: pci12: <ACPI PCI bus> on pcib12
Jul 8 20:02:30 mybox kernel: pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
Jul 8 20:02:30 mybox kernel: pci13: <ACPI PCI bus> on pcib13
Jul 8 20:02:30 mybox kernel: pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
Jul 8 20:02:30 mybox kernel: pci14: <ACPI PCI bus> on pcib14
Jul 8 20:02:30 mybox kernel: pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
Jul 8 20:02:30 mybox kernel: pci15: <ACPI PCI bus> on pcib15
Jul 8 20:02:30 mybox kernel: pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
Jul 8 20:02:30 mybox kernel: pci16: <ACPI PCI bus> on pcib16
Jul 8 20:02:30 mybox kernel: pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
Jul 8 20:02:30 mybox kernel: pci17: <ACPI PCI bus> on pcib17
Jul 8 20:02:30 mybox kernel: pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
Jul 8 20:02:30 mybox kernel: pci18: <ACPI PCI bus> on pcib18
Jul 8 20:02:30 mybox kernel: pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
Jul 8 20:02:30 mybox kernel: pci19: <ACPI PCI bus> on pcib19
Jul 8 20:02:30 mybox kernel: pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
Jul 8 20:02:30 mybox kernel: pci20: <ACPI PCI bus> on pcib20
Jul 8 20:02:30 mybox kernel: pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
Jul 8 20:02:30 mybox kernel: pci21: <ACPI PCI bus> on pcib21
Jul 8 20:02:30 mybox kernel: pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
Jul 8 20:02:30 mybox kernel: pci22: <ACPI PCI bus> on pcib22
Jul 8 20:02:30 mybox kernel: pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
Jul 8 20:02:30 mybox kernel: pci23: <ACPI PCI bus> on pcib23
Jul 8 20:02:30 mybox kernel: pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
Jul 8 20:02:30 mybox kernel: pci24: <ACPI PCI bus> on pcib24
Jul 8 20:02:30 mybox kernel: pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
Jul 8 20:02:30 mybox kernel: pci25: <ACPI PCI bus> on pcib25
Jul 8 20:02:30 mybox kernel: pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
Jul 8 20:02:30 mybox kernel: pci26: <ACPI PCI bus> on pcib26
Jul 8 20:02:30 mybox kernel: pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
Jul 8 20:02:30 mybox kernel: pci27: <ACPI PCI bus> on pcib27
Jul 8 20:02:30 mybox kernel: pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
Jul 8 20:02:30 mybox kernel: pci28: <ACPI PCI bus> on pcib28
Jul 8 20:02:30 mybox kernel: pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
Jul 8 20:02:30 mybox kernel: pci29: <ACPI PCI bus> on pcib29
Jul 8 20:02:30 mybox kernel: pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
Jul 8 20:02:30 mybox kernel: pci30: <ACPI PCI bus> on pcib30
Jul 8 20:02:30 mybox kernel: pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
Jul 8 20:02:30 mybox kernel: pci31: <ACPI PCI bus> on pcib31
Jul 8 20:02:30 mybox kernel: pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
Jul 8 20:02:30 mybox kernel: pci32: <ACPI PCI bus> on pcib32
Jul 8 20:02:30 mybox kernel: pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
Jul 8 20:02:30 mybox kernel: pci33: <ACPI PCI bus> on pcib33
Jul 8 20:02:30 mybox kernel: pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
Jul 8 20:02:30 mybox kernel: pci34: <ACPI PCI bus> on pcib34
Jul 8 20:02:30 mybox kernel: acpi_acad0: <AC Adapter> on acpi0
Jul 8 20:02:30 mybox kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Jul 8 20:02:30 mybox kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Jul 8 20:02:30 mybox kernel: kbd0 at atkbd0
Jul 8 20:02:30 mybox kernel: atkbd0: [GIANT-LOCKED]
Jul 8 20:02:30 mybox kernel: psm0: <PS/2 Mouse> irq 12 on atkbdc0
Jul 8 20:02:30 mybox kernel: psm0: [GIANT-LOCKED]
Jul 8 20:02:30 mybox kernel: psm0: model IntelliMouse, device ID 3
Jul 8 20:02:30 mybox kernel: fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Jul 8 20:02:30 mybox kernel: fd0: <1440-KB 3.5" drive> on fdc0 drive 0
Jul 8 20:02:30 mybox kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
Jul 8 20:02:30 mybox kernel: sc0: <System console> at flags 0x100 on isa0
Jul 8 20:02:30 mybox kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Jul 8 20:02:30 mybox kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Jul 8 20:02:30 mybox kernel: ppc0: cannot reserve I/O port range
Jul 8 20:02:30 mybox kernel: Timecounters tick every 1.000 msec
Jul 8 20:02:30 mybox kernel: em0: link state changed to UP
Jul 8 20:02:30 mybox kernel: random: unblocking device.
Jul 8 20:02:30 mybox kernel: cd0 at ata1 bus 0 scbus1 target 0 lun 0
Jul 8 20:02:30 mybox kernel: cd0: <NECVMWar VMware IDE CDR10 1.00> Removable CD-ROM SCSI device
Jul 8 20:02:30 mybox kernel: cd0: Serial Number 10000000000000000001
Jul 8 20:02:30 mybox kernel: cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
Jul 8 20:02:30 mybox kernel: cd0: Attempt to query device size failed: NOT READY, Medium not present
Jul 8 20:02:30 mybox kernel: cd0: quirks=0x40<RETRY_BUSY>
Jul 8 20:02:30 mybox kernel: da0 at mpt0 bus 0 scbus2 target 0 lun 0
Jul 8 20:02:30 mybox kernel: da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
Jul 8 20:02:30 mybox kernel: da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
Jul 8 20:02:30 mybox kernel: da0: Command Queueing enabled
Jul 8 20:02:30 mybox kernel: da0: 12288MB (25165824 512 byte sectors)
Jul 8 20:02:30 mybox kernel: da0: quirks=0x40<RETRY_BUSY>
Jul 8 20:02:30 mybox kernel: SMP: AP CPU #1 Launched!
Jul 8 20:02:30 mybox kernel: SMP: AP CPU #2 Launched!
Jul 8 20:02:30 mybox kernel: SMP: AP CPU #3 Launched!
Jul 8 20:02:30 mybox kernel: Trying to mount root from ufs:/dev/da0p2 [rw]...
Jul 8 20:02:30 mybox kernel: WARNING: / was not properly dismounted
Jul 8 20:02:30 mybox kernel: vmx0: link state changed to UP
Jul 8 20:02:30 mybox kernel: vmx1: link state changed to UP
Jul 8 20:02:30 mybox kernel: Limiting icmp unreach response from 2687 to 200 packets/sec
Jul 8 20:02:30 mybox kernel: Limiting icmp unreach response from 2254 to 200 packets/sec
Jul 8 20:02:30 mybox kernel: ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to deny, logging disabled
Jul 8 20:02:30 mybox kernel: DUMMYNET 0 with IPv6 initialized (100409)
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched FIFO loaded
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched QFQ loaded
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched RR loaded
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched WF2Q+ loaded
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched PRIO loaded
Jul 8 20:02:30 mybox kernel: Limiting icmp unreach response from 2374 to 200 packets/sec
Jul 8 20:02:30 mybox kernel: Limiting icmp unreach response from 2602 to 200 packets/sec
Jul 8 20:02:31 mybox kernel: Limiting icmp unreach response from 2283 to 200 packets/sec
Jul 8 20:02:32 mybox kernel: Limiting icmp unreach response from 2333 to 200 packets/sec
Jul 8 20:02:33 mybox kernel: Limiting icmp unreach response from 2217 to 200 packets/sec
Jul 8 20:02:34 mybox kernel: Limiting icmp unreach response from 2134 to 200 packets/sec
Jul 8 20:02:35 mybox kernel: Limiting icmp unreach response from 2064 to 200 packets/sec
Jul 8 20:02:36 mybox kernel: Limiting icmp unreach response from 1662 to 200 packets/sec
Jul 8 20:02:37 mybox kernel: Limiting icmp unreach response from 1704 to 200 packets/sec
Jul 8 20:02:38 mybox kernel: Limiting icmp unreach response from 1632 to 200 packets/sec
Jul 8 20:02:39 mybox kernel: Limiting icmp unreach response from 1472 to 200 packets/sec
Jul 8 20:02:41 mybox kernel: Limiting icmp unreach response from 1494 to 200 packets/sec
Jul 8 20:02:42 mybox kernel: Limiting icmp unreach response from 1474 to 200 packets/sec
Jul 8 20:02:43 mybox kernel: Limiting icmp unreach response from 1520 to 200 packets/sec
Jul 8 20:02:44 mybox kernel: Limiting icmp unreach response from 1409 to 200 packets/sec
Jul 8 20:02:45 mybox kernel: Limiting icmp unreach response from 1227 to 200 packets/sec
Jul 8 20:02:46 mybox kernel: Limiting icmp unreach response from 1264 to 200 packets/sec
Jul 8 20:02:47 mybox kernel: Limiting icmp unreach response from 1105 to 200 packets/sec
Jul 8 20:02:48 mybox kernel: Limiting icmp unreach response from 1049 to 200 packets/sec
Jul 8 20:02:49 mybox kernel: Limiting icmp unreach response from 971 to 200 packets/sec
Jul 8 20:02:50 mybox kernel: Limiting icmp unreach response from 882 to 200 packets/sec
Jul 8 20:02:51 mybox kernel: Limiting icmp unreach response from 906 to 200 packets/sec
Jul 8 20:02:52 mybox kernel: Limiting icmp unreach response from 826 to 200 packets/sec
Jul 8 20:02:53 mybox kernel: Limiting icmp unreach response from 721 to 200 packets/sec
Jul 8 20:02:54 mybox kernel: Limiting icmp unreach response from 666 to 200 packets/sec
Jul 8 20:02:55 mybox kernel: Limiting icmp unreach response from 582 to 200 packets/sec
Jul 8 20:02:56 mybox kernel: Limiting icmp unreach response from 551 to 200 packets/sec
Jul 8 20:02:57 mybox kernel: Limiting icmp unreach response from 514 to 200 packets/sec
Jul 8 20:02:57 mybox kernel: .
Jul 8 20:02:57 mybox kernel: WARNING: attempt to domain_add(netgraph) after domainfinalize()
Jul 8 20:02:58 mybox kernel: ipfw: rule 50: setsockopt(IP_FW_DEL): Invalid argument
Jul 8 20:02:58 mybox kernel: done
Jul 8 20:02:59 mybox kernel:
Jul 8 20:02:59 mybox kernel: VMware memory control driver initialized
Jul 8 20:02:59 mybox kernel: done
============================
This is a new project replacing a cisco 7206, 700-sessions 800mbit/s
I am not familiar with troubleshooting kernel panic's,
I am suspecting that the crash is happening inside the netgraph module because the crash is happening at the
instruction pointer = 0x20:0xffffffff81c38283
I included the 2 two crash logs. I need some help to to figure out what to do next.
-Dbaud
============================
The box is a:
# uname -a
FreeBSD mybox.example.com 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016 ***@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
# kldstat
Id Refs Address Size Name
1 34 0xffffffff80200000 17bc6a8 kernel
2 2 0xffffffff81c11000 114db ipfw.ko
3 1 0xffffffff81c23000 d32f dummynet.ko
4 1 0xffffffff81c31000 3831 ng_socket.ko
5 9 0xffffffff81c35000 ba02 netgraph.ko
6 1 0xffffffff81c41000 2b99 ng_mppc.ko
7 1 0xffffffff81c44000 80c rc4.ko
8 1 0xffffffff81c45000 23dc vmmemctl.ko
9 1 0xffffffff81c48000 397d ng_l2tp.ko
10 1 0xffffffff81c4c000 4b04 ng_ksocket.ko
11 1 0xffffffff81c51000 17d6 ng_tee.ko
12 1 0xffffffff81c53000 40d2 ng_iface.ko
13 1 0xffffffff81c58000 5829 ng_ppp.ko
14 1 0xffffffff81c5e000 18b1 ng_tcpmss.ko
15 1 0xffffffff81c60000 2df7 ng_vjc.ko
=======================================
First crash dump:
Jul 8 08:09:04 mybox syslogd: kernel boot file is /boot/kernel/kernel
Jul 8 08:09:04 mybox kernel:
Jul 8 08:09:04 mybox kernel:
Jul 8 08:09:04 mybox kernel: Fatal trap 12: page fault while in kernel mode
Jul 8 08:09:04 mybox kernel: cpuid = 1; apic id = 01
Jul 8 08:09:04 mybox kernel: fault virtual address = 0x28
Jul 8 08:09:04 mybox kernel: fault code = supervisor read data, page not present
Jul 8 08:09:04 mybox kernel: instruction pointer = 0x20:0xffffffff81c38283
Jul 8 08:09:04 mybox kernel: stack pointer = 0x28:0xfffffe0174d85540
Jul 8 08:09:04 mybox kernel: frame pointer = 0x28:0xfffffe0174d85580
Jul 8 08:09:04 mybox kernel: code segment = base 0x0, limit 0xfffff, type 0x1b
Jul 8 08:09:04 mybox kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Jul 8 08:09:04 mybox kernel: processor eflags = interrupt enabled, resume, IOPL = 0
Jul 8 08:09:04 mybox kernel: current process = 628 (ng_queue2)
Jul 8 08:09:04 mybox kernel: trap number = 12
Jul 8 08:09:04 mybox kernel: panic: page fault
Jul 8 08:09:04 mybox kernel: cpuid = 1
Jul 8 08:09:04 mybox kernel: KDB: stack backtrace:
Jul 8 08:09:04 mybox kernel: #0 0xffffffff8098e390 at kdb_backtrace+0x60
Jul 8 08:09:04 mybox kernel: #1 0xffffffff80951066 at vpanic+0x126
Jul 8 08:09:04 mybox kernel: #2 0xffffffff80950f33 at panic+0x43
Jul 8 08:09:04 mybox kernel: #3 0xffffffff80d55f7b at trap_fatal+0x36b
Jul 8 08:09:04 mybox kernel: #4 0xffffffff80d5627d at trap_pfault+0x2ed
Jul 8 08:09:04 mybox kernel: #5 0xffffffff80d558fa at trap+0x47a
Jul 8 08:09:04 mybox kernel: #6 0xffffffff80d3b8d2 at calltrap+0x8
Jul 8 08:09:04 mybox kernel: #7 0xffffffff81c5e509 at ng_tcpmss_rcvdata+0x2d9
Jul 8 08:09:04 mybox kernel: #8 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 08:09:04 mybox kernel: #9 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 08:09:04 mybox kernel: #10 0xffffffff81c5a1c8 at ng_ppp_comp_recv+0x148
Jul 8 08:09:04 mybox kernel: #11 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 08:09:04 mybox kernel: #12 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 08:09:04 mybox kernel: #13 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 08:09:04 mybox kernel: #14 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 08:09:04 mybox kernel: #15 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 08:09:04 mybox kernel: #16 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 08:09:04 mybox kernel: #17 0xffffffff81c4d3e2 at ng_ksocket_incoming2+0x2f2
Jul 8 08:09:04 mybox kernel: Uptime: 5d17h47m38s
Jul 8 08:09:04 mybox kernel: Copyright (c) 1992-2016 The FreeBSD Project.
Jul 8 08:09:04 mybox kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jul 8 08:09:04 mybox kernel: The Regents of the University of California. All rights reserved.
Jul 8 08:09:04 mybox kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jul 8 08:09:04 mybox kernel: FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016
Jul 8 08:09:04 mybox kernel: ***@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
Jul 8 08:09:04 mybox kernel: FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Jul 8 08:09:04 mybox kernel: CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.00-MHz K8-class CPU)
Jul 8 08:09:04 mybox kernel: Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7
Jul 8 08:09:04 mybox kernel: Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
Jul 8 08:09:04 mybox kernel: Features2=0x9fba2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
Jul 8 08:09:04 mybox kernel: AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Jul 8 08:09:04 mybox kernel: AMD Features2=0x1<LAHF>
Jul 8 08:09:04 mybox kernel: Structured Extended Features=0x2<TSCADJ>
Jul 8 08:09:04 mybox kernel: TSC: P-state invariant
Jul 8 08:09:04 mybox kernel: Hypervisor: Origin = "VMwareVMware"
Jul 8 08:09:04 mybox kernel: real memory = 6442450944 (6144 MB)
Jul 8 08:09:04 mybox kernel: avail memory = 6183518208 (5897 MB)
Jul 8 08:09:04 mybox kernel: Event timer "LAPIC" quality 600
Jul 8 08:09:04 mybox kernel: ACPI APIC Table: <PTLTD APIC >
Jul 8 08:09:04 mybox kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Jul 8 08:09:04 mybox kernel: FreeBSD/SMP: 1 package(s) x 4 core(s)
Jul 8 08:09:04 mybox kernel: cpu0 (BSP): APIC ID: 0
Jul 8 08:09:04 mybox kernel: cpu1 (AP): APIC ID: 1
Jul 8 08:09:04 mybox kernel: cpu2 (AP): APIC ID: 2
Jul 8 08:09:04 mybox kernel: cpu3 (AP): APIC ID: 3
Jul 8 08:09:04 mybox kernel: random: <Software, Yarrow> initialized
Jul 8 08:09:04 mybox kernel: MADT: Forcing active-low polarity and level trigger for SCI
Jul 8 08:09:04 mybox kernel: ioapic0 <Version 1.1> irqs 0-23 on motherboard
Jul 8 08:09:04 mybox kernel: kbd1 at kbdmux0
Jul 8 08:09:04 mybox kernel: acpi0: <INTEL 440BX> on motherboard
Jul 8 08:09:04 mybox kernel: acpi0: Power Button (fixed)
Jul 8 08:09:04 mybox kernel: Timecounter "HPET" frequency 14318180 Hz quality 950
Jul 8 08:09:04 mybox kernel: cpu0: <ACPI CPU> on acpi0
Jul 8 08:09:04 mybox kernel: cpu1: <ACPI CPU> on acpi0
Jul 8 08:09:04 mybox kernel: cpu2: <ACPI CPU> on acpi0
Jul 8 08:09:04 mybox kernel: cpu3: <ACPI CPU> on acpi0
Jul 8 08:09:04 mybox kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Jul 8 08:09:04 mybox kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Jul 8 08:09:04 mybox kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Jul 8 08:09:04 mybox kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Jul 8 08:09:04 mybox kernel: Event timer "RTC" frequency 32768 Hz quality 0
Jul 8 08:09:04 mybox kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Jul 8 08:09:04 mybox kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
Jul 8 08:09:04 mybox kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Jul 8 08:09:04 mybox kernel: pci0: <ACPI PCI bus> on pcib0
Jul 8 08:09:04 mybox kernel: pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
Jul 8 08:09:04 mybox kernel: pci1: <ACPI PCI bus> on pcib1
Jul 8 08:09:04 mybox kernel: isab0: <PCI-ISA bridge> at device 7.0 on pci0
Jul 8 08:09:04 mybox kernel: isa0: <ISA bus> on isab0
Jul 8 08:09:04 mybox kernel: atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
Jul 8 08:09:04 mybox kernel: ata0: <ATA channel> at channel 0 on atapci0
Jul 8 08:09:04 mybox kernel: ata1: <ATA channel> at channel 1 on atapci0
Jul 8 08:09:04 mybox kernel: pci0: <bridge> at device 7.3 (no driver attached)
Jul 8 08:09:04 mybox kernel: vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
Jul 8 08:09:04 mybox kernel: vgapci0: Boot video device
Jul 8 08:09:04 mybox kernel: mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0
Jul 8 08:09:04 mybox kernel: mpt0: MPI Version=1.2.0.0
Jul 8 08:09:04 mybox kernel: pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
Jul 8 08:09:04 mybox kernel: pci2: <ACPI PCI bus> on pcib2
Jul 8 08:09:04 mybox kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x2000-0x203f mem 0xfd5c0000-0xfd5dffff,0xfdff0000-0xfdffffff irq 18 at device 0.0 on pci2
Jul 8 08:09:04 mybox kernel: em0: Ethernet address: 00:50:56:8d:d7:57
Jul 8 08:09:04 mybox kernel: pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
Jul 8 08:09:04 mybox kernel: pci3: <ACPI PCI bus> on pcib3
Jul 8 08:09:04 mybox kernel: vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd4fc000-0xfd4fcfff,0xfd4fd000-0xfd4fdfff,0xfd4fe000-0xfd4fffff irq 18 at device 0.0 on pci3
Jul 8 08:09:04 mybox kernel: vmx0: Ethernet address: 00:50:56:8d:4c:68
Jul 8 08:09:04 mybox kernel: pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
Jul 8 08:09:04 mybox kernel: pci4: <ACPI PCI bus> on pcib4
Jul 8 08:09:04 mybox kernel: pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
Jul 8 08:09:04 mybox kernel: pci5: <ACPI PCI bus> on pcib5
Jul 8 08:09:04 mybox kernel: pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
Jul 8 08:09:04 mybox kernel: pci6: <ACPI PCI bus> on pcib6
Jul 8 08:09:04 mybox kernel: pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
Jul 8 08:09:04 mybox kernel: pci7: <ACPI PCI bus> on pcib7
Jul 8 08:09:04 mybox kernel: pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
Jul 8 08:09:04 mybox kernel: pci8: <ACPI PCI bus> on pcib8
Jul 8 08:09:04 mybox kernel: pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
Jul 8 08:09:04 mybox kernel: pci9: <ACPI PCI bus> on pcib9
Jul 8 08:09:04 mybox kernel: pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
Jul 8 08:09:04 mybox kernel: pci10: <ACPI PCI bus> on pcib10
Jul 8 08:09:04 mybox kernel: pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
Jul 8 08:09:04 mybox kernel: pci11: <ACPI PCI bus> on pcib11
Jul 8 08:09:04 mybox kernel: vmx1: <VMware VMXNET3 Ethernet Adapter> port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci11
Jul 8 08:09:04 mybox kernel: vmx1: Ethernet address: 00:50:56:8d:85:29
Jul 8 08:09:04 mybox kernel: pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
Jul 8 08:09:04 mybox kernel: pci12: <ACPI PCI bus> on pcib12
Jul 8 08:09:04 mybox kernel: pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
Jul 8 08:09:04 mybox kernel: pci13: <ACPI PCI bus> on pcib13
Jul 8 08:09:04 mybox kernel: pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
Jul 8 08:09:04 mybox kernel: pci14: <ACPI PCI bus> on pcib14
Jul 8 08:09:04 mybox kernel: pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
Jul 8 08:09:04 mybox kernel: pci15: <ACPI PCI bus> on pcib15
Jul 8 08:09:04 mybox kernel: pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
Jul 8 08:09:04 mybox kernel: pci16: <ACPI PCI bus> on pcib16
Jul 8 08:09:04 mybox kernel: pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
Jul 8 08:09:04 mybox kernel: pci17: <ACPI PCI bus> on pcib17
Jul 8 08:09:04 mybox kernel: pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
Jul 8 08:09:04 mybox kernel: pci18: <ACPI PCI bus> on pcib18
Jul 8 08:09:04 mybox kernel: pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
Jul 8 08:09:04 mybox kernel: pci19: <ACPI PCI bus> on pcib19
Jul 8 08:09:04 mybox kernel: pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
Jul 8 08:09:04 mybox kernel: pci20: <ACPI PCI bus> on pcib20
Jul 8 08:09:04 mybox kernel: pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
Jul 8 08:09:04 mybox kernel: pci21: <ACPI PCI bus> on pcib21
Jul 8 08:09:04 mybox kernel: pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
Jul 8 08:09:04 mybox kernel: pci22: <ACPI PCI bus> on pcib22
Jul 8 08:09:04 mybox kernel: pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
Jul 8 08:09:04 mybox kernel: pci23: <ACPI PCI bus> on pcib23
Jul 8 08:09:04 mybox kernel: pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
Jul 8 08:09:04 mybox kernel: pci24: <ACPI PCI bus> on pcib24
Jul 8 08:09:04 mybox kernel: pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
Jul 8 08:09:04 mybox kernel: pci25: <ACPI PCI bus> on pcib25
Jul 8 08:09:04 mybox kernel: pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
Jul 8 08:09:04 mybox kernel: pci26: <ACPI PCI bus> on pcib26
Jul 8 08:09:04 mybox kernel: pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
Jul 8 08:09:04 mybox kernel: pci27: <ACPI PCI bus> on pcib27
Jul 8 08:09:04 mybox kernel: pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
Jul 8 08:09:04 mybox kernel: pci28: <ACPI PCI bus> on pcib28
Jul 8 08:09:04 mybox kernel: pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
Jul 8 08:09:04 mybox kernel: pci29: <ACPI PCI bus> on pcib29
Jul 8 08:09:04 mybox kernel: pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
Jul 8 08:09:04 mybox kernel: pci30: <ACPI PCI bus> on pcib30
Jul 8 08:09:04 mybox kernel: pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
Jul 8 08:09:04 mybox kernel: pci31: <ACPI PCI bus> on pcib31
Jul 8 08:09:04 mybox kernel: pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
Jul 8 08:09:04 mybox kernel: pci32: <ACPI PCI bus> on pcib32
Jul 8 08:09:04 mybox kernel: pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
Jul 8 08:09:04 mybox kernel: pci33: <ACPI PCI bus> on pcib33
Jul 8 08:09:04 mybox kernel: pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
Jul 8 08:09:04 mybox kernel: pci34: <ACPI PCI bus> on pcib34
Jul 8 08:09:04 mybox kernel: acpi_acad0: <AC Adapter> on acpi0
Jul 8 08:09:04 mybox kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Jul 8 08:09:04 mybox kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Jul 8 08:09:04 mybox kernel: kbd0 at atkbd0
Jul 8 08:09:04 mybox kernel: atkbd0: [GIANT-LOCKED]
Jul 8 08:09:04 mybox kernel: psm0: <PS/2 Mouse> irq 12 on atkbdc0
Jul 8 08:09:04 mybox kernel: psm0: [GIANT-LOCKED]
Jul 8 08:09:04 mybox kernel: psm0: model IntelliMouse, device ID 3
Jul 8 08:09:04 mybox kernel: fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Jul 8 08:09:04 mybox kernel: fd0: <1440-KB 3.5" drive> on fdc0 drive 0
Jul 8 08:09:04 mybox kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
Jul 8 08:09:04 mybox kernel: sc0: <System console> at flags 0x100 on isa0
Jul 8 08:09:04 mybox kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Jul 8 08:09:04 mybox kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Jul 8 08:09:04 mybox kernel: ppc0: cannot reserve I/O port range
Jul 8 08:09:04 mybox kernel: Timecounters tick every 1.000 msec
Jul 8 08:09:04 mybox kernel: em0: link state changed to UP
Jul 8 08:09:04 mybox kernel: random: unblocking device.
Jul 8 08:09:04 mybox kernel: cd0 at ata1 bus 0 scbus1 target 0 lun 0
Jul 8 08:09:04 mybox kernel: cd0: <NECVMWar VMware IDE CDR10 1.00> Removable CD-ROM SCSI device
Jul 8 08:09:04 mybox kernel: cd0: Serial Number 10000000000000000001
Jul 8 08:09:04 mybox kernel: cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
Jul 8 08:09:04 mybox kernel: cd0: Attempt to query device size failed: NOT READY, Medium not present
Jul 8 08:09:04 mybox kernel: cd0: quirks=0x40<RETRY_BUSY>
Jul 8 08:09:04 mybox kernel: da0 at mpt0 bus 0 scbus2 target 0 lun 0
Jul 8 08:09:04 mybox kernel: da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
Jul 8 08:09:04 mybox kernel: da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
Jul 8 08:09:04 mybox kernel: da0: Command Queueing enabled
Jul 8 08:09:04 mybox kernel: da0: 12288MB (25165824 512 byte sectors)
Jul 8 08:09:04 mybox kernel: da0: quirks=0x40<RETRY_BUSY>
Jul 8 08:09:04 mybox kernel: SMP: AP CPU #3 Launched!
Jul 8 08:09:04 mybox kernel: SMP: AP CPU #1 Launched!
Jul 8 08:09:04 mybox kernel: SMP: AP CPU #2 Launched!
Jul 8 08:09:04 mybox kernel: Trying to mount root from ufs:/dev/da0p2 [rw]...
Jul 8 08:09:04 mybox kernel: WARNING: / was not properly dismounted
Jul 8 08:09:04 mybox kernel: vmx0: link state changed to UP
Jul 8 08:09:04 mybox kernel: vmx1: link state changed to UP
Jul 8 08:09:04 mybox kernel: Limiting icmp unreach response from 1816 to 200 packets/sec
Jul 8 08:09:04 mybox kernel: Limiting icmp unreach response from 1982 to 200 packets/sec
Jul 8 08:09:04 mybox kernel: ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to deny, logging disabled
Jul 8 08:09:04 mybox kernel: DUMMYNET 0 with IPv6 initialized (100409)
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched FIFO loaded
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched QFQ loaded
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched RR loaded
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched WF2Q+ loaded
Jul 8 08:09:04 mybox kernel: load_dn_sched dn_sched PRIO loaded
Jul 8 08:09:04 mybox kernel: Limiting icmp unreach response from 2125 to 200 packets/sec
Jul 8 08:09:05 mybox kernel: WARNING: attempt to domain_add(netgraph) after domainfinalize()
Jul 8 08:09:06 mybox kernel: done
Jul 8 08:09:07 mybox kernel: VMware memory control driver initialized
Jul 8 08:09:07 mybox kernel: done
===========================================
Second crash dump
Jul 8 20:02:30 mybox syslogd: kernel boot file is /boot/kernel/kernel
Jul 8 20:02:30 mybox kernel:
Jul 8 20:02:30 mybox kernel:
Jul 8 20:02:30 mybox kernel: Fatal trap 12: page fault while in kernel mode
Jul 8 20:02:30 mybox kernel: cpuid = 2; apic id = 02
Jul 8 20:02:30 mybox kernel: fault virtual address = 0x28
Jul 8 20:02:30 mybox kernel: fault code = supervisor read data, page not present
Jul 8 20:02:30 mybox kernel: instruction pointer = 0x20:0xffffffff81c38283
Jul 8 20:02:30 mybox kernel: stack pointer = 0x28:0xfffffe0174d99660
Jul 8 20:02:30 mybox kernel: frame pointer = 0x28:0xfffffe0174d996a0
Jul 8 20:02:30 mybox kernel: code segment = base 0x0, limit 0xfffff, type 0x1b
Jul 8 20:02:30 mybox kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Jul 8 20:02:30 mybox kernel: processor eflags = interrupt enabled, resume, IOPL = 0
Jul 8 20:02:30 mybox kernel: current process = 627 (ng_queue3)
Jul 8 20:02:30 mybox kernel: trap number = 12
Jul 8 20:02:30 mybox kernel: panic: page fault
Jul 8 20:02:30 mybox kernel: cpuid = 2
Jul 8 20:02:30 mybox kernel: KDB: stack backtrace:
Jul 8 20:02:30 mybox kernel: #0 0xffffffff8098e390 at kdb_backtrace+0x60
Jul 8 20:02:30 mybox kernel: #1 0xffffffff80951066 at vpanic+0x126
Jul 8 20:02:30 mybox kernel: #2 0xffffffff80950f33 at panic+0x43
Jul 8 20:02:30 mybox kernel: #3 0xffffffff80d55f7b at trap_fatal+0x36b
Jul 8 20:02:30 mybox kernel: #4 0xffffffff80d5627d at trap_pfault+0x2ed
Jul 8 20:02:30 mybox kernel: #5 0xffffffff80d558fa at trap+0x47a
Jul 8 20:02:30 mybox kernel: #6 0xffffffff80d3b8d2 at calltrap+0x8
Jul 8 20:02:30 mybox kernel: #7 0xffffffff81c5a042 at ng_ppp_bypass+0x62
Jul 8 20:02:30 mybox kernel: #8 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 20:02:30 mybox kernel: #9 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 20:02:30 mybox kernel: #10 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 20:02:30 mybox kernel: #11 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 20:02:30 mybox kernel: #12 0xffffffff81c370ca at ng_apply_item+0x21a
Jul 8 20:02:30 mybox kernel: #13 0xffffffff81c36d1a at ng_snd_item+0x38a
Jul 8 20:02:30 mybox kernel: #14 0xffffffff81c4d3e2 at ng_ksocket_incoming2+0x2f2
Jul 8 20:02:30 mybox kernel: #15 0xffffffff81c36f62 at ng_apply_item+0xb2
Jul 8 20:02:30 mybox kernel: #16 0xffffffff81c38d39 at ngthread+0x1b9
Jul 8 20:02:30 mybox kernel: #17 0xffffffff8091a4ea at fork_exit+0x9a
Jul 8 20:02:30 mybox kernel: Uptime: 11h52m56s
Jul 8 20:02:30 mybox kernel: Dumping 591 out of 6119 MB:..3%..11%..22%..33%..41%..52%..63%..71%..82%..92%
Jul 8 20:02:30 mybox kernel: Dump complete
Jul 8 20:02:30 mybox kernel: Copyright (c) 1992-2016 The FreeBSD Project.
Jul 8 20:02:30 mybox kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jul 8 20:02:30 mybox kernel: The Regents of the University of California. All rights reserved.
Jul 8 20:02:30 mybox kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jul 8 20:02:30 mybox kernel: FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016
Jul 8 20:02:30 mybox kernel: ***@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
Jul 8 20:02:30 mybox kernel: FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Jul 8 20:02:30 mybox kernel: CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.00-MHz K8-class CPU)
Jul 8 20:02:30 mybox kernel: Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7
Jul 8 20:02:30 mybox kernel: Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
Jul 8 20:02:30 mybox kernel: Features2=0x9fba2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
Jul 8 20:02:30 mybox kernel: AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Jul 8 20:02:30 mybox kernel: AMD Features2=0x1<LAHF>
Jul 8 20:02:30 mybox kernel: Structured Extended Features=0x2<TSCADJ>
Jul 8 20:02:30 mybox kernel: TSC: P-state invariant
Jul 8 20:02:30 mybox kernel: Hypervisor: Origin = "VMwareVMware"
Jul 8 20:02:30 mybox kernel: real memory = 6442450944 (6144 MB)
Jul 8 20:02:30 mybox kernel: avail memory = 6183518208 (5897 MB)
Jul 8 20:02:30 mybox kernel: Event timer "LAPIC" quality 600
Jul 8 20:02:30 mybox kernel: ACPI APIC Table: <PTLTD APIC >
Jul 8 20:02:30 mybox kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Jul 8 20:02:30 mybox kernel: FreeBSD/SMP: 1 package(s) x 4 core(s)
Jul 8 20:02:30 mybox kernel: cpu0 (BSP): APIC ID: 0
Jul 8 20:02:30 mybox kernel: cpu1 (AP): APIC ID: 1
Jul 8 20:02:30 mybox kernel: cpu2 (AP): APIC ID: 2
Jul 8 20:02:30 mybox kernel: cpu3 (AP): APIC ID: 3
Jul 8 20:02:30 mybox kernel: random: <Software, Yarrow> initialized
Jul 8 20:02:30 mybox kernel: MADT: Forcing active-low polarity and level trigger for SCI
Jul 8 20:02:30 mybox kernel: ioapic0 <Version 1.1> irqs 0-23 on motherboard
Jul 8 20:02:30 mybox kernel: kbd1 at kbdmux0
Jul 8 20:02:30 mybox kernel: acpi0: <INTEL 440BX> on motherboard
Jul 8 20:02:30 mybox kernel: acpi0: Power Button (fixed)
Jul 8 20:02:30 mybox kernel: Timecounter "HPET" frequency 14318180 Hz quality 950
Jul 8 20:02:30 mybox kernel: cpu0: <ACPI CPU> on acpi0
Jul 8 20:02:30 mybox kernel: cpu1: <ACPI CPU> on acpi0
Jul 8 20:02:30 mybox kernel: cpu2: <ACPI CPU> on acpi0
Jul 8 20:02:30 mybox kernel: cpu3: <ACPI CPU> on acpi0
Jul 8 20:02:30 mybox kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Jul 8 20:02:30 mybox kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Jul 8 20:02:30 mybox kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Jul 8 20:02:30 mybox kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Jul 8 20:02:30 mybox kernel: Event timer "RTC" frequency 32768 Hz quality 0
Jul 8 20:02:30 mybox kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Jul 8 20:02:30 mybox kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
Jul 8 20:02:30 mybox kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Jul 8 20:02:30 mybox kernel: pci0: <ACPI PCI bus> on pcib0
Jul 8 20:02:30 mybox kernel: pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
Jul 8 20:02:30 mybox kernel: pci1: <ACPI PCI bus> on pcib1
Jul 8 20:02:30 mybox kernel: isab0: <PCI-ISA bridge> at device 7.0 on pci0
Jul 8 20:02:30 mybox kernel: isa0: <ISA bus> on isab0
Jul 8 20:02:30 mybox kernel: atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
Jul 8 20:02:30 mybox kernel: ata0: <ATA channel> at channel 0 on atapci0
Jul 8 20:02:30 mybox kernel: ata1: <ATA channel> at channel 1 on atapci0
Jul 8 20:02:30 mybox kernel: pci0: <bridge> at device 7.3 (no driver attached)
Jul 8 20:02:30 mybox kernel: vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
Jul 8 20:02:30 mybox kernel: vgapci0: Boot video device
Jul 8 20:02:30 mybox kernel: mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0
Jul 8 20:02:30 mybox kernel: mpt0: MPI Version=1.2.0.0
Jul 8 20:02:30 mybox kernel: pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
Jul 8 20:02:30 mybox kernel: pci2: <ACPI PCI bus> on pcib2
Jul 8 20:02:30 mybox kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x2000-0x203f mem 0xfd5c0000-0xfd5dffff,0xfdff0000-0xfdffffff irq 18 at device 0.0 on pci2
Jul 8 20:02:30 mybox kernel: em0: Ethernet address: 00:50:56:8d:d7:57
Jul 8 20:02:30 mybox kernel: pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
Jul 8 20:02:30 mybox kernel: pci3: <ACPI PCI bus> on pcib3
Jul 8 20:02:30 mybox kernel: vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd4fc000-0xfd4fcfff,0xfd4fd000-0xfd4fdfff,0xfd4fe000-0xfd4fffff irq 18 at device 0.0 on pci3
Jul 8 20:02:30 mybox kernel: vmx0: Ethernet address: 00:50:56:8d:4c:68
Jul 8 20:02:30 mybox kernel: pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
Jul 8 20:02:30 mybox kernel: pci4: <ACPI PCI bus> on pcib4
Jul 8 20:02:30 mybox kernel: pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
Jul 8 20:02:30 mybox kernel: pci5: <ACPI PCI bus> on pcib5
Jul 8 20:02:30 mybox kernel: pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
Jul 8 20:02:30 mybox kernel: pci6: <ACPI PCI bus> on pcib6
Jul 8 20:02:30 mybox kernel: pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
Jul 8 20:02:30 mybox kernel: pci7: <ACPI PCI bus> on pcib7
Jul 8 20:02:30 mybox kernel: pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
Jul 8 20:02:30 mybox kernel: pci8: <ACPI PCI bus> on pcib8
Jul 8 20:02:30 mybox kernel: pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
Jul 8 20:02:30 mybox kernel: pci9: <ACPI PCI bus> on pcib9
Jul 8 20:02:30 mybox kernel: pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
Jul 8 20:02:30 mybox kernel: pci10: <ACPI PCI bus> on pcib10
Jul 8 20:02:30 mybox kernel: pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
Jul 8 20:02:30 mybox kernel: pci11: <ACPI PCI bus> on pcib11
Jul 8 20:02:30 mybox kernel: vmx1: <VMware VMXNET3 Ethernet Adapter> port 0x5000-0x500f mem 0xfd3fc000-0xfd3fcfff,0xfd3fd000-0xfd3fdfff,0xfd3fe000-0xfd3fffff irq 19 at device 0.0 on pci11
Jul 8 20:02:30 mybox kernel: vmx1: Ethernet address: 00:50:56:8d:85:29
Jul 8 20:02:30 mybox kernel: pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
Jul 8 20:02:30 mybox kernel: pci12: <ACPI PCI bus> on pcib12
Jul 8 20:02:30 mybox kernel: pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
Jul 8 20:02:30 mybox kernel: pci13: <ACPI PCI bus> on pcib13
Jul 8 20:02:30 mybox kernel: pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
Jul 8 20:02:30 mybox kernel: pci14: <ACPI PCI bus> on pcib14
Jul 8 20:02:30 mybox kernel: pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
Jul 8 20:02:30 mybox kernel: pci15: <ACPI PCI bus> on pcib15
Jul 8 20:02:30 mybox kernel: pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
Jul 8 20:02:30 mybox kernel: pci16: <ACPI PCI bus> on pcib16
Jul 8 20:02:30 mybox kernel: pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
Jul 8 20:02:30 mybox kernel: pci17: <ACPI PCI bus> on pcib17
Jul 8 20:02:30 mybox kernel: pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
Jul 8 20:02:30 mybox kernel: pci18: <ACPI PCI bus> on pcib18
Jul 8 20:02:30 mybox kernel: pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
Jul 8 20:02:30 mybox kernel: pci19: <ACPI PCI bus> on pcib19
Jul 8 20:02:30 mybox kernel: pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
Jul 8 20:02:30 mybox kernel: pci20: <ACPI PCI bus> on pcib20
Jul 8 20:02:30 mybox kernel: pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
Jul 8 20:02:30 mybox kernel: pci21: <ACPI PCI bus> on pcib21
Jul 8 20:02:30 mybox kernel: pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
Jul 8 20:02:30 mybox kernel: pci22: <ACPI PCI bus> on pcib22
Jul 8 20:02:30 mybox kernel: pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
Jul 8 20:02:30 mybox kernel: pci23: <ACPI PCI bus> on pcib23
Jul 8 20:02:30 mybox kernel: pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
Jul 8 20:02:30 mybox kernel: pci24: <ACPI PCI bus> on pcib24
Jul 8 20:02:30 mybox kernel: pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
Jul 8 20:02:30 mybox kernel: pci25: <ACPI PCI bus> on pcib25
Jul 8 20:02:30 mybox kernel: pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
Jul 8 20:02:30 mybox kernel: pci26: <ACPI PCI bus> on pcib26
Jul 8 20:02:30 mybox kernel: pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
Jul 8 20:02:30 mybox kernel: pci27: <ACPI PCI bus> on pcib27
Jul 8 20:02:30 mybox kernel: pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
Jul 8 20:02:30 mybox kernel: pci28: <ACPI PCI bus> on pcib28
Jul 8 20:02:30 mybox kernel: pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
Jul 8 20:02:30 mybox kernel: pci29: <ACPI PCI bus> on pcib29
Jul 8 20:02:30 mybox kernel: pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
Jul 8 20:02:30 mybox kernel: pci30: <ACPI PCI bus> on pcib30
Jul 8 20:02:30 mybox kernel: pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
Jul 8 20:02:30 mybox kernel: pci31: <ACPI PCI bus> on pcib31
Jul 8 20:02:30 mybox kernel: pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
Jul 8 20:02:30 mybox kernel: pci32: <ACPI PCI bus> on pcib32
Jul 8 20:02:30 mybox kernel: pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
Jul 8 20:02:30 mybox kernel: pci33: <ACPI PCI bus> on pcib33
Jul 8 20:02:30 mybox kernel: pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
Jul 8 20:02:30 mybox kernel: pci34: <ACPI PCI bus> on pcib34
Jul 8 20:02:30 mybox kernel: acpi_acad0: <AC Adapter> on acpi0
Jul 8 20:02:30 mybox kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Jul 8 20:02:30 mybox kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Jul 8 20:02:30 mybox kernel: kbd0 at atkbd0
Jul 8 20:02:30 mybox kernel: atkbd0: [GIANT-LOCKED]
Jul 8 20:02:30 mybox kernel: psm0: <PS/2 Mouse> irq 12 on atkbdc0
Jul 8 20:02:30 mybox kernel: psm0: [GIANT-LOCKED]
Jul 8 20:02:30 mybox kernel: psm0: model IntelliMouse, device ID 3
Jul 8 20:02:30 mybox kernel: fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Jul 8 20:02:30 mybox kernel: fd0: <1440-KB 3.5" drive> on fdc0 drive 0
Jul 8 20:02:30 mybox kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
Jul 8 20:02:30 mybox kernel: sc0: <System console> at flags 0x100 on isa0
Jul 8 20:02:30 mybox kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Jul 8 20:02:30 mybox kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Jul 8 20:02:30 mybox kernel: ppc0: cannot reserve I/O port range
Jul 8 20:02:30 mybox kernel: Timecounters tick every 1.000 msec
Jul 8 20:02:30 mybox kernel: em0: link state changed to UP
Jul 8 20:02:30 mybox kernel: random: unblocking device.
Jul 8 20:02:30 mybox kernel: cd0 at ata1 bus 0 scbus1 target 0 lun 0
Jul 8 20:02:30 mybox kernel: cd0: <NECVMWar VMware IDE CDR10 1.00> Removable CD-ROM SCSI device
Jul 8 20:02:30 mybox kernel: cd0: Serial Number 10000000000000000001
Jul 8 20:02:30 mybox kernel: cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
Jul 8 20:02:30 mybox kernel: cd0: Attempt to query device size failed: NOT READY, Medium not present
Jul 8 20:02:30 mybox kernel: cd0: quirks=0x40<RETRY_BUSY>
Jul 8 20:02:30 mybox kernel: da0 at mpt0 bus 0 scbus2 target 0 lun 0
Jul 8 20:02:30 mybox kernel: da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
Jul 8 20:02:30 mybox kernel: da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
Jul 8 20:02:30 mybox kernel: da0: Command Queueing enabled
Jul 8 20:02:30 mybox kernel: da0: 12288MB (25165824 512 byte sectors)
Jul 8 20:02:30 mybox kernel: da0: quirks=0x40<RETRY_BUSY>
Jul 8 20:02:30 mybox kernel: SMP: AP CPU #1 Launched!
Jul 8 20:02:30 mybox kernel: SMP: AP CPU #2 Launched!
Jul 8 20:02:30 mybox kernel: SMP: AP CPU #3 Launched!
Jul 8 20:02:30 mybox kernel: Trying to mount root from ufs:/dev/da0p2 [rw]...
Jul 8 20:02:30 mybox kernel: WARNING: / was not properly dismounted
Jul 8 20:02:30 mybox kernel: vmx0: link state changed to UP
Jul 8 20:02:30 mybox kernel: vmx1: link state changed to UP
Jul 8 20:02:30 mybox kernel: Limiting icmp unreach response from 2687 to 200 packets/sec
Jul 8 20:02:30 mybox kernel: Limiting icmp unreach response from 2254 to 200 packets/sec
Jul 8 20:02:30 mybox kernel: ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to deny, logging disabled
Jul 8 20:02:30 mybox kernel: DUMMYNET 0 with IPv6 initialized (100409)
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched FIFO loaded
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched QFQ loaded
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched RR loaded
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched WF2Q+ loaded
Jul 8 20:02:30 mybox kernel: load_dn_sched dn_sched PRIO loaded
Jul 8 20:02:30 mybox kernel: Limiting icmp unreach response from 2374 to 200 packets/sec
Jul 8 20:02:30 mybox kernel: Limiting icmp unreach response from 2602 to 200 packets/sec
Jul 8 20:02:31 mybox kernel: Limiting icmp unreach response from 2283 to 200 packets/sec
Jul 8 20:02:32 mybox kernel: Limiting icmp unreach response from 2333 to 200 packets/sec
Jul 8 20:02:33 mybox kernel: Limiting icmp unreach response from 2217 to 200 packets/sec
Jul 8 20:02:34 mybox kernel: Limiting icmp unreach response from 2134 to 200 packets/sec
Jul 8 20:02:35 mybox kernel: Limiting icmp unreach response from 2064 to 200 packets/sec
Jul 8 20:02:36 mybox kernel: Limiting icmp unreach response from 1662 to 200 packets/sec
Jul 8 20:02:37 mybox kernel: Limiting icmp unreach response from 1704 to 200 packets/sec
Jul 8 20:02:38 mybox kernel: Limiting icmp unreach response from 1632 to 200 packets/sec
Jul 8 20:02:39 mybox kernel: Limiting icmp unreach response from 1472 to 200 packets/sec
Jul 8 20:02:41 mybox kernel: Limiting icmp unreach response from 1494 to 200 packets/sec
Jul 8 20:02:42 mybox kernel: Limiting icmp unreach response from 1474 to 200 packets/sec
Jul 8 20:02:43 mybox kernel: Limiting icmp unreach response from 1520 to 200 packets/sec
Jul 8 20:02:44 mybox kernel: Limiting icmp unreach response from 1409 to 200 packets/sec
Jul 8 20:02:45 mybox kernel: Limiting icmp unreach response from 1227 to 200 packets/sec
Jul 8 20:02:46 mybox kernel: Limiting icmp unreach response from 1264 to 200 packets/sec
Jul 8 20:02:47 mybox kernel: Limiting icmp unreach response from 1105 to 200 packets/sec
Jul 8 20:02:48 mybox kernel: Limiting icmp unreach response from 1049 to 200 packets/sec
Jul 8 20:02:49 mybox kernel: Limiting icmp unreach response from 971 to 200 packets/sec
Jul 8 20:02:50 mybox kernel: Limiting icmp unreach response from 882 to 200 packets/sec
Jul 8 20:02:51 mybox kernel: Limiting icmp unreach response from 906 to 200 packets/sec
Jul 8 20:02:52 mybox kernel: Limiting icmp unreach response from 826 to 200 packets/sec
Jul 8 20:02:53 mybox kernel: Limiting icmp unreach response from 721 to 200 packets/sec
Jul 8 20:02:54 mybox kernel: Limiting icmp unreach response from 666 to 200 packets/sec
Jul 8 20:02:55 mybox kernel: Limiting icmp unreach response from 582 to 200 packets/sec
Jul 8 20:02:56 mybox kernel: Limiting icmp unreach response from 551 to 200 packets/sec
Jul 8 20:02:57 mybox kernel: Limiting icmp unreach response from 514 to 200 packets/sec
Jul 8 20:02:57 mybox kernel: .
Jul 8 20:02:57 mybox kernel: WARNING: attempt to domain_add(netgraph) after domainfinalize()
Jul 8 20:02:58 mybox kernel: ipfw: rule 50: setsockopt(IP_FW_DEL): Invalid argument
Jul 8 20:02:58 mybox kernel: done
Jul 8 20:02:59 mybox kernel:
Jul 8 20:02:59 mybox kernel: VMware memory control driver initialized
Jul 8 20:02:59 mybox kernel: done
============================