Recent Changes for "System Info" - Wiki Spot

System Info

Graham.Freeman — 2009-08-24 03:41:22

+info from site news

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 33:	Line 33:
	+ [[anchor(2009-8-23)]] + ''2009-8-23'' Wikispot sysadmins addressed system performance issues by terminating a hung database process and improving the way the webserver handles requests. The hung database process resulted in approximately 5 minutes of downtime, and approximately 1 hour of sub-par performance. - ["Users/PhilipNeustrom" Philip], [wiki:davis:Users/AmitVainsencher Amit] and ["Users/Graham.Freeman" Graham] (with troubleshooting assistance from ["Users/JasonAller" Jason]) +
Line 37:	Line 40:
-

System Info

Graham.Freeman — 2009-08-08 09:58:18

+IRC, for full geekiness

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 6:	Line 6:
	+ + Also, you can join us in ["IRC"] in channel #wikispot on server irc.freenode.net. However, if we're busy resolving an outage, we're probably not going to be very chatty!

System Info

Graham.Freeman — 2009-08-08 08:23:18

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 34:	Line 34:
- We updated a number of packages to slightly newer point releases, including python, PostgreSQL, and the kernel. We also changed the network configuration so as to clearly separate Wiki Spot's billable network traffic from Amit's billable network traffic. This resulting in an initial outage of approximately 30 seconds (while PostgreSQL restarted) and a second outage of approximately 2 minutes (while the server restarted). Things went very smoothly. - [wiki:davis:Users/AmitVainsencher Amit] and ["Users/Graham.Freeman" Graham]	+ We updated a number of packages to slightly newer point releases, including python, PostgreSQL, and the kernel. We also changed the network configuration so as to clearly separate Wiki Spot's billable network traffic from Amit's billable network traffic. This resulted in an initial outage of approximately 30 seconds (while PostgreSQL restarted) and a second outage of approximately 2 minutes (while the server restarted). Things went very smoothly. - [wiki:davis:Users/AmitVainsencher Amit] and ["Users/Graham.Freeman" Graham]

System Info

Graham.Freeman — 2009-08-08 08:21:20

+twitterrific

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 2:	Line 2:
	+ + = How can I get status updates if this site is offline? = + + You can follow [http://twitter.com/wikispot_status @wikispot_status] on Twitter.

System Info

Graham.Freeman — 2009-08-08 07:36:37

+details on tonight's maintenance

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 21:	Line 21:
- We also have a another VPS dedicated to software development and testing. All three servers are colocated in the [http://cernio.com/ Cernio Tech Co-op]'s cabinets at United Layer's facility in the 200 Paul datacenter in San Francisco, California. In addition, we have access on short notice to server capacity in London.	+ We also have a another VPS dedicated to software development and testing. All three servers are colocated in the [http://cernio.com/ Cernio Tech Co-op]'s cabinets at United Layer's facility in the 200 Paul datacenter in San Francisco, California. In addition, we have access on short notice to server capacity in London, Minneapolis, and Santa Clara.
Line 26:	Line 26:
	+ + [[anchor(2009-8-8)]] + == OS maintenance - 8 Aug 2009 == + + We updated a number of packages to slightly newer point releases, including python, PostgreSQL, and the kernel. We also changed the network configuration so as to clearly separate Wiki Spot's billable network traffic from Amit's billable network traffic. This resulting in an initial outage of approximately 30 seconds (while PostgreSQL restarted) and a second outage of approximately 2 minutes (while the server restarted). Things went very smoothly. - [wiki:davis:Users/AmitVainsencher Amit] and ["Users/Graham.Freeman" Graham] +

System Info

AmitVainsencher — 2009-03-10 10:34:24

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 19:	Line 19:
- ''Yep, we're using the 8.04 server edition'' -[wiki:davis:Users/AmitVainsencher AV]	+ ''Yep, we're using the 8.04 server edition. I used a [http://jailtime.org/ jailtime] tarball for Leo, and the standard 8.04.2 server edition CD to install onto Baxter'' -[wiki:davis:Users/AmitVainsencher AV]

System Info

AmitVainsencher — 2009-03-10 10:29:17

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 19:	Line 19:
	+ ''Yep, we're using the 8.04 server edition'' -[wiki:davis:Users/AmitVainsencher AV]

System Info

JabberWokky — 2009-03-10 10:12:40

Active support rather than passive.

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 18:	Line 18:
	+ ''If it matters, Amit, I didn't protest because I very much supported the decision (there just wasn't much to add because, as you say, nobody really objected). I've found the LTS series to be quite excellent: stable and well crafted. I had earlier assumed you had meant to use the Ubuntu Server Edition... did you?'' --["Users/JabberWokky"]

System Info

AmitVainsencher — 2009-03-10 09:52:56

runaway italics!

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 17:	Line 17:
- ''The choice was mainly due to Graham's suggestion, and the fact that nobody else protested. Personally, I like Ubuntu because of the fact that it has a much more recent "stable" release than Debian, and that it's already on all of the servers I maintain at work... it gives me a single security mailing list to watch for all of my sysadmin hats. Ubuntu 8.04 also has recent enough versions of all the software needed to run Sycamore available via apt (except a tiny module, python-memcached) (upon checking, this appears to be true for Debian stable/lenny as well). -[wiki:davis:Users/AmitVainsencher AV]	+ ''The choice was mainly due to Graham's suggestion, and the fact that nobody else protested. Personally, I like Ubuntu because of the fact that it has a much more recent "stable" release than Debian, and that it's already on all of the servers I maintain at work... it gives me a single security mailing list to watch for all of my sysadmin hats. Ubuntu 8.04 also has recent enough versions of all the software needed to run Sycamore available via apt (except a tiny module, python-memcached) (upon checking, this appears to be true for Debian stable/lenny as well).'' -[wiki:davis:Users/AmitVainsencher AV]

System Info

AmitVainsencher — 2009-03-10 09:51:34

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 17:	Line 17:
- ''The choice was mainly due to Graham's suggestion, and the fact that nobody else protested. Personally, I like Ubuntu because of the fact that it has a much more recent "stable" release than Debian, and that it's already on all of the servers I maintain at work... it gives me a single security mailing list to watch for all of my sysadmin hats. Ubuntu 8.04 also has recent enough versions of all the software needed to run Sycamore available via apt (except a tiny module, python-memcached), whereas I don't believe Debian stable does. -[wiki:davis:Users/AmitVainsencher AV]	+ ''The choice was mainly due to Graham's suggestion, and the fact that nobody else protested. Personally, I like Ubuntu because of the fact that it has a much more recent "stable" release than Debian, and that it's already on all of the servers I maintain at work... it gives me a single security mailing list to watch for all of my sysadmin hats. Ubuntu 8.04 also has recent enough versions of all the software needed to run Sycamore available via apt (except a tiny module, python-memcached) (upon checking, this appears to be true for Debian stable/lenny as well). -[wiki:davis:Users/AmitVainsencher AV]

System Info

AmitVainsencher — 2009-03-10 09:46:03

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 17:	Line 17:
	+ ''The choice was mainly due to Graham's suggestion, and the fact that nobody else protested. Personally, I like Ubuntu because of the fact that it has a much more recent "stable" release than Debian, and that it's already on all of the servers I maintain at work... it gives me a single security mailing list to watch for all of my sysadmin hats. Ubuntu 8.04 also has recent enough versions of all the software needed to run Sycamore available via apt (except a tiny module, python-memcached), whereas I don't believe Debian stable does. -[wiki:davis:Users/AmitVainsencher AV]

System Info

WilliamLewis — 2009-03-10 07:03:42

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 16:	Line 16:
	+ ''Both ["Users/NickSchmalenberger" Nick] and ["Users/WilliamLewis" I] are wondering why Ubuntu is being chosen over Debian. Not that it's a wrong decision... we're just curious what motivated it.''

System Info

AmitVainsencher — 2009-03-10 06:44:12

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 15:	Line 15:
- The former production server, Baxter, has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/product_barebones_detail.aspx?pid=37 Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. We're planning to move primary serving responsibilities back to Baxter once we purchase some new hard drives for it.	+ The former production server, Baxter, has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/product_barebones_detail.aspx?pid=37 Tyan Transport] barebone package and used to run [http://www.gentoo.org/ Gentoo Linux]. We're planning to move primary serving responsibilities back to Baxter once we purchase some new hard drives for it, and replace Gentoo with [http://www.ubuntu.com/ Ubuntu] 8.04 LTS.

System Info

AmitVainsencher — 2009-03-10 06:40:02

geeky server details

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 13:	Line 13:
- We presently operate on a single dedicated production server and two development servers - a separate physical server and a virtual private server. All three are colocated in [http://cernio.com/ Cernio Tech Co-op]'s cabinets at United Layer's facility in the 200 Paul datacenter in San Francisco, California.	+ As of Mar 7, 2009, we are running on a Xen virtual private server (VPS) named Leo, on Amit's personal hardware. The system has two Opteron 275's (2.2GHz), for a total of four processor cores, 8GB of DDR1-400MHz ECC/Registered memory, and a four drive 1.5TB hardware RAID5 running atop a 3ware 9550SXU (with a battery backup for the write cache). Leo has 4GB of memory, two processor cores, and 300GB of storage to itself.
Line 15:	Line 15:
- The production server has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/product_barebones_detail.aspx?pid=37 Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux].	+ The former production server, Baxter, has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/product_barebones_detail.aspx?pid=37 Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. We're planning to move primary serving responsibilities back to Baxter once we purchase some new hard drives for it.
Line 17:	Line 17:
- We have access on short notice to server capacity in London. -	+ We also have a another VPS dedicated to software development and testing. All three servers are colocated in the [http://cernio.com/ Cernio Tech Co-op]'s cabinets at United Layer's facility in the 200 Paul datacenter in San Francisco, California. In addition, we have access on short notice to server capacity in London.

System Info

AmitVainsencher — 2009-03-09 00:22:18

I guess asterisks need spaces on both sides to show up as a bullet?

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 29:	Line 29:
- * Amit moved his personal server downstairs first. Once he verified that it was in working order after the move, Philip made wikispot read-only and then performed a database dump. - * Amit's personal server is configured with Xen, and he provisioned a virtual machine named leo, running Ubuntu 8.04 LTS, with resources comparable to those of baxter. He also copied over the Sycamore configuration/setup ahead of time, so that all that was needed for leo to take over wikispot responsibilities was a database restore. - * After bringing up the VPS leo, they moved the wikispot database dump over to it and loaded it up. Everything worked as expected, and they silently switched IP addresses between leo and baxter. - * After leo had taken over for wikispot, the site had write access restored to the general public. Baxter was shut off, brought downstairs and racked. Amit then took two new hard drives that had been donated by Graham and installed them into baxter as its new primary drives. He installed Ubuntu 8.04 LTS on them, while leaving the old Gentoo drives drives untouched. - * At this moment, wikispot is being served from the VPS leo until a later time when we will transition the services back over to baxter. - * This whole process took about 9 hours (from roughly 5PM-2AM (technically until 3AM if you count the daylight savings switch that happened at 2AM)). - * '''Many thanks to Graham and the other Cernio people''' that took time to help us out despite the insanity of trying to get the rest of Cernio in order before the move-out deadline. This would've been a hell of a lot more difficult without your hard work and assistance!	+ * Amit moved his personal server downstairs first. Once he verified that it was in working order after the move, Philip made wikispot read-only and then performed a database dump. + * Amit's personal server is configured with Xen, and he provisioned a virtual machine named leo, running Ubuntu 8.04 LTS, with resources comparable to those of baxter. He also copied over the Sycamore configuration/setup ahead of time, so that all that was needed for leo to take over wikispot responsibilities was a database restore. + * After bringing up the VPS leo, they moved the wikispot database dump over to it and loaded it up. Everything worked as expected, and they silently switched IP addresses between leo and baxter. + * After leo had taken over for wikispot, the site had write access restored to the general public. Baxter was shut off, brought downstairs and racked. Amit then took two new hard drives that had been donated by Graham and installed them into baxter as its new primary drives. He installed Ubuntu 8.04 LTS on them, while leaving the old Gentoo drives drives untouched. + * At this moment, wikispot is being served from the VPS leo until a later time when we will transition the services back over to baxter. + * This whole process took about 9 hours (from roughly 5PM-2AM (technically until 3AM if you count the daylight savings switch that happened at 2AM)). + * '''Many thanks to Graham and the other Cernio people''' that took time to help us out despite the insanity of trying to get the rest of Cernio in order before the move-out deadline. This would've been a hell of a lot more difficult without your hard work and assistance!

System Info

AmitVainsencher — 2009-03-09 00:05:08

update move info

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 26:	Line 26:
- The weekend of Mar 7, Wikispot's host Cernio scheduled to move the entirety of its two cabinets on the third floor of 200 Paul into three cabinets on the first floor in a brand new data room. In addition to having a ton of free space, this new room also theoretically won't suffer from the cooling problems that plagued the old room. At the same time, our primary server baxter was in dire need of an OS replacement because its install of Gentoo had become unmaintainable due to unpredictable behavior during routine upgrades. Additionally, the SQL database was heavily bloated due to our having set the Postgres max_fsm_pages too low. This was causing postgres to "forget" about regions of the database that needed to be garbage collected. The database bloat was the main reason for the progressive slowdown we've seen in the past few months.	+ The weekend of Mar 7, Wikispot's host Cernio had scheduled to move the entirety of its two cabinets on the third floor of 200 Paul into three cabinets on the first floor in a brand new data room. In addition to having a ton of free space, this new room also theoretically wouldn't suffer from the cooling problems that plagued the old room. At the same time, our primary server baxter was in dire need of an OS replacement because its install of Gentoo had become unmaintainable due to unpredictable behavior during routine upgrades. Additionally, the SQL database was heavily bloated due to our having left the Postgres max_fsm_pages parameter at its default value, which was way too low. This was causing postgres to "forget" about regions of the database that needed to be garbage collected, and as a result it was growing without bound: a freshly restored copy only consumed about 13GB, whereas the current one was nearly 90GB. The database bloat was the main reason for the progressive slowdown we've seen in the past few months.
Line 30:	Line 30:
- * Amit's personal server is configured with Xen, and he provisioned a virtual machine named leo, running Ubuntu 8.04 LTS, with resources comparable to those of baxter. He also copied over the Sycamore configuration/setup over ahead of time, so that all that was needed for leo to take over wikispot responsibilities was a database restore.	+ * Amit's personal server is configured with Xen, and he provisioned a virtual machine named leo, running Ubuntu 8.04 LTS, with resources comparable to those of baxter. He also copied over the Sycamore configuration/setup ahead of time, so that all that was needed for leo to take over wikispot responsibilities was a database restore.
Line 35:	Line 35:
	+ * '''Many thanks to Graham and the other Cernio people''' that took time to help us out despite the insanity of trying to get the rest of Cernio in order before the move-out deadline. This would've been a hell of a lot more difficult without your hard work and assistance!

System Info

PhilipNeustrom — 2009-03-08 12:21:24

anchor

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 24:	Line 24:
	+ [[anchor(2009-3-7)]]

System Info

AmitVainsencher — 2009-03-08 12:18:21

Summary of today's sysadmin work

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 22:	Line 22:
- = Recent Hardware Maintenance =	+ = Recent Maintenance = + + == Big move/Reorganization - Mar 7 2009 == + The weekend of Mar 7, Wikispot's host Cernio scheduled to move the entirety of its two cabinets on the third floor of 200 Paul into three cabinets on the first floor in a brand new data room. In addition to having a ton of free space, this new room also theoretically won't suffer from the cooling problems that plagued the old room. At the same time, our primary server baxter was in dire need of an OS replacement because its install of Gentoo had become unmaintainable due to unpredictable behavior during routine upgrades. Additionally, the SQL database was heavily bloated due to our having set the Postgres max_fsm_pages too low. This was causing postgres to "forget" about regions of the database that needed to be garbage collected. The database bloat was the main reason for the progressive slowdown we've seen in the past few months. + + Amit, Philip, and Graham tackled all of these issues simultaneously in the following way: + * Amit moved his personal server downstairs first. Once he verified that it was in working order after the move, Philip made wikispot read-only and then performed a database dump. + * Amit's personal server is configured with Xen, and he provisioned a virtual machine named leo, running Ubuntu 8.04 LTS, with resources comparable to those of baxter. He also copied over the Sycamore configuration/setup over ahead of time, so that all that was needed for leo to take over wikispot responsibilities was a database restore. + * After bringing up the VPS leo, they moved the wikispot database dump over to it and loaded it up. Everything worked as expected, and they silently switched IP addresses between leo and baxter. + * After leo had taken over for wikispot, the site had write access restored to the general public. Baxter was shut off, brought downstairs and racked. Amit then took two new hard drives that had been donated by Graham and installed them into baxter as its new primary drives. He installed Ubuntu 8.04 LTS on them, while leaving the old Gentoo drives drives untouched. + * At this moment, wikispot is being served from the VPS leo until a later time when we will transition the services back over to baxter. + * This whole process took about 9 hours (from roughly 5PM-2AM (technically until 3AM if you count the daylight savings switch that happened at 2AM)).

System Info

Graham.Freeman — 2009-02-01 00:02:13

work done

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 23:	Line 23:
	+ + == Network maintenance - Jan 2009 == + + I activated an additional network interface on the production server. -Graham (31 Jan 09 at about 15:30 California time)

System Info

NickSchmalenberger — 2009-01-28 13:39:06

Revert to version 33 (weirdos).

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
- [[Image(War_Room.jpg, "The top secret wiki control center located hundreds of feet underground in a nuclear-safe bunker.", 500, thumbnail)]]	+ [[Image(War_Room.jpg, "The top secret wiki control center located hundreds of feet underground in a nuclear-safe bunker.*", 500, thumbnail)]]
Line 118:	Line 118:
	+ * Not really. :) It's actually [wiki:WikiPedia:LIGO LIGO]

System Info

2009-01-28 13:15:58

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
- [[Image(War_Room.jpg, "The top secret wiki control center located hundreds of feet underground in a nuclear-safe bunker.*", 500, thumbnail)]]	+ [[Image(War_Room.jpg, "The top secret wiki control center located hundreds of feet underground in a nuclear-safe bunker.", 500, thumbnail)]]
Line 118:	Line 118:
- * Not really. :) It's actually [wiki:WikiPedia:LIGO LIGO]

System Info

NickSchmalenberger — 2008-07-06 05:10:50

link to kernel.org

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 115:	Line 115:
- The Wiki Spot project wouldn't be possible without the awesomeness that is: [http://www.gnu.org/ GNU/Linux], [http://python.org Python], [http://www.postgresql.org/ PostgreSQL], [http://xapian.org Xapian], [http://www.danga.com/memcached/ memcached], and [http://www.lighttpd.net/ lighttpd].	+ The Wiki Spot project wouldn't be possible without the awesomeness that is: [http://www.gnu.org/ GNU]/[http://www.kernel.org/ Linux], [http://python.org Python], [http://www.postgresql.org/ PostgreSQL], [http://xapian.org Xapian], [http://www.danga.com/memcached/ memcached], and [http://www.lighttpd.net/ lighttpd].

System Info

AmitVainsencher — 2007-11-15 21:49:37

fix hardware link

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 15:	Line 15:
- The production server has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux].	+ The production server has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/product_barebones_detail.aspx?pid=37 Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux].

System Info

PhilipNeustrom — 2007-08-23 00:56:09

power-by type thing

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 114:	Line 114:
	+ = Thanks to... = + The Wiki Spot project wouldn't be possible without the awesomeness that is: [http://www.gnu.org/ GNU/Linux], [http://python.org Python], [http://www.postgresql.org/ PostgreSQL], [http://xapian.org Xapian], [http://www.danga.com/memcached/ memcached], and [http://www.lighttpd.net/ lighttpd]. +

System Info

GrahamFreeman — 2007-07-30 04:45:43

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 17:	Line 17:
- We have access on short notice to server capacity in London, and will soon have similar access in New York.	+ We have access on short notice to server capacity in London.

System Info

2007-05-03 01:35:08

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 94:	Line 94:
- = What are some statistics on the site =	+ = What are some statistics on the site? =

System Info

2007-05-03 01:35:00

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 94:	Line 94:
- = What are some statistics on the site? =	+ = What are some statistics on the site =

System Info

GrahamFreeman — 2007-04-04 04:43:50

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 80:	Line 80:
- The transition went relatively painlessly. First, Amit made the wiki read-only, to prevent possible data loss. Then, he dumped the database and all files associated with the wiki to a system already in the new data center in SF. This was done to ensure that the site could be quickly restored if the primary server was damaged or destroyed in transit. After backing up, Graham and Amit pulled the server and loaded it into the back of Amit's car, and proceeded to drive to San Francisco (approximately 1AM). Fortunately, the backup precautions proved unnecessary, as the server was still alive and kicking on arrival. Once Graham and Amit got the server running at the new location, Graham transitioned the DNS for all of the system's domains (see above for a list) to the server's new IP address, and configured a web traffic proxy from the system's old IP in Santa Rosa to the new one in San Francisco. This was done to ensure no traffic would be lost while the new DNS information was propagating (DNS changes can take anywhere from 30 minutes to 24 hours to fully propogate).	+ + The transition went relatively painlessly. First, Amit made the wiki read-only, to prevent possible data loss. Then, he dumped the database and all files associated with the wiki to a system already in the new data center in SF. This was done to ensure that the site could be quickly restored if the primary server was damaged or destroyed in transit. After backing up, Graham and Amit pulled the server and loaded it into the back of Amit's car, and proceeded to drive to San Francisco (approximately 1AM). Fortunately, the backup precautions proved unnecessary, as the server was still alive and kicking on arrival. + + Once Graham and Amit got the server running at the new location, Graham transitioned the DNS for all of the system's domains (see above for a list) to the server's new IP address, and configured a web traffic proxy from the system's old IP in Santa Rosa to the new one in San Francisco. This was done to ensure no traffic would be lost while the new DNS information was propagating (DNS changes can take anywhere from 30 minutes to 24 hours to fully propagate). +

System Info

GrahamFreeman — 2007-04-04 04:43:28

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 79:	Line 79:
- In the early morning of Oct 22, 2006, ["Users/GrahamFreeman"] and ["Users/AmitVainsencher"] moved DavisWiki's server between [wiki:davis:"Cernio"] cabinets at [http://www.sonic.net/ Sonic.net] in Santa Rosa, and [http://www.unitedlayer.com/ United Layer]'s facility at 200 Paul Ave, San Francisco. This move was in line with Cernio's overall transition to United Layer as its colocation provider. The transition was made because the new data center offered better connectivity (for example, to countries in Eastern Europe and the Middle East) coupled with nearly a 50% reduction in bandwidth costs.	+ In the early morning of Oct 22, 2006, ["Users/GrahamFreeman"] and [wiki:davis:Users/AmitVainsencher] moved DavisWiki's server between [wiki:davis:"Cernio"] cabinets at [http://www.sonic.net/ Sonic.net] in Santa Rosa, and [http://www.unitedlayer.com/ United Layer]'s facility at 200 Paul Ave, San Francisco. This move was in line with Cernio's overall transition to United Layer as its colocation provider. The transition was made because the new data center offered better connectivity (for example, to countries in Eastern Europe and the Middle East) coupled with nearly a 50% reduction in bandwidth costs.

System Info

GrahamFreeman — 2007-04-04 04:43:02

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 9:	Line 9:
- ["wiki:davis:Users/AmitVainsencher" Amit Vainsencher] and ["Users/GrahamFreeman" Graham Freeman]	+ [wiki:davis:Users/AmitVainsencher Amit Vainsencher] and ["Users/GrahamFreeman" Graham Freeman]

System Info

GrahamFreeman — 2007-04-04 04:42:51

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 9:	Line 9:
- ["Users/AmitVainsencher" Amit Vainsencher] and ["Users/GrahamFreeman" Graham Freeman]	+ ["wiki:davis:Users/AmitVainsencher" Amit Vainsencher] and ["Users/GrahamFreeman" Graham Freeman]

System Info

GrahamFreeman — 2007-04-04 02:24:49

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 108:	Line 108:
- You may be interested in the ["Wiki Community/Technical Discussion"].	+ You may be interested in the [wiki:davis:"Wiki Community/Technical Discussion"].

System Info

GrahamFreeman — 2007-04-04 02:24:35

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 87:	Line 87:
- *Why do you think, that DavisWiki is the world's best city wiki? What makes it best? --["Users/WilhelmBuehler"]	+ *Why do you think, that DavisWiki is the world's best city wiki? What makes it best? --[wiki:davis:"Users/WilhelmBuehler"]

System Info

GrahamFreeman — 2007-04-04 02:24:24

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 84:	Line 84:
- * Are we big in Eastern Europe and the Middle East? -["Users/MattJurach"]	+ * Are we big in Eastern Europe and the Middle East? -[wiki:davis:"Users/MattJurach"]

System Info

GrahamFreeman — 2007-04-04 02:24:14

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 83:	Line 83:
- * Well done, gentlemen. -["Users/JimStewart"]	+ * Well done, gentlemen. -[wiki:davis:"Users/JimStewart"]

System Info

GrahamFreeman — 2007-04-04 02:23:59

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 79:	Line 79:
- In the early morning of Oct 22, 2006, ["Users/GrahamFreeman"] and ["Users/AmitVainsencher"] moved DavisWiki's server between ["Cernio"] cabinets at [http://www.sonic.net/ Sonic.net] in Santa Rosa, and [http://www.unitedlayer.com/ United Layer]'s facility at 200 Paul Ave, San Francisco. This move was in line with Cernio's overall transition to United Layer as its colocation provider. The transition was made because the new data center offered better connectivity (for example, to countries in Eastern Europe and the Middle East) coupled with nearly a 50% reduction in bandwidth costs.	+ In the early morning of Oct 22, 2006, ["Users/GrahamFreeman"] and ["Users/AmitVainsencher"] moved DavisWiki's server between [wiki:davis:"Cernio"] cabinets at [http://www.sonic.net/ Sonic.net] in Santa Rosa, and [http://www.unitedlayer.com/ United Layer]'s facility at 200 Paul Ave, San Francisco. This move was in line with Cernio's overall transition to United Layer as its colocation provider. The transition was made because the new data center offered better connectivity (for example, to countries in Eastern Europe and the Middle East) coupled with nearly a 50% reduction in bandwidth costs.

System Info

GrahamFreeman — 2007-04-04 02:23:29

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 20:	Line 20:
- See also the page on [wiki:daviswiki:"Wiki hosting"].	+ See also the page on [wiki:davis:"Wiki hosting"].

System Info

GrahamFreeman — 2007-04-04 02:23:21

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 20:	Line 20:
- See also the page on ["Wiki hosting"].	+ See also the page on [wiki:daviswiki:"Wiki hosting"].

System Info

GrahamFreeman — 2007-04-04 00:54:43

minor reformat; add info on last night's downtime

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 9:	Line 9:
- ["AmitVainsencher" Amit Vainsencher] and ["GrahamFreeman" Graham Freeman]	+ ["Users/AmitVainsencher" Amit Vainsencher] and ["Users/GrahamFreeman" Graham Freeman]
Line 16:	Line 16:
-
Line 25:	Line 24:
	+ == Network maintenance - April 2007 == + + What: At about 22:30 (UTC-7) on Mon 2 April 07, all Wiki Spot-hosted wikis went offline and remained offline for approximately 20 minutes, when service was restored at 22:50 (UTC-7). + + Why: This was the result of two things: (1) ["Users/GrahamFreeman" my] incomplete understanding of how Gentoo Linux init scripts work, and (2) Gentoo's use of an alpha-quality init script for the network functionality. + + Background: + + In coordination with ["Users/PhilipNeustrom" Philip Neustrom], ["Users/GrahamFreeman" I] began the process of switching Wiki Spot's main production server over to a new IP address in accordance with a migration process we agreed to with our colocation vendor. (Paraphrasing: Graham: "This should incur no downtime, but just in case, when can we afford 20 minutes of downtime?" Philip: "Actually, right now would be best." Graham: "OK, I'll get started.") + + As per the documentation at [http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=4&chap=2] I started this by replacing the following lines in /etc/conf.d/net : + + config_eth0=( "207.7.145.200 netmask 255.255.255.192 brd 207.7.145.255" ) + routes_eth0=( "default gw 207.7.145.193" ) + + with these: + + config_eth0=( + "209.237.247.200/24" + "207.7.145.200 netmask 255.255.255.192 brd 207.7.145.255" + ) + routes_eth0=( "default gw 209.237.247.1" ) + + I then mis-read a section at [http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=4] to mean that I could safely run "/etc/init.d/net restart". Unfortunately, while the "start()" function is required, the "restart()" function is not. Furthermore, the script lacked sanity checking sufficient to warn me that I passed an invalid argument, and apparently instead just ran its usual "start()" function despite the fact that the network and dependent services (including the various hosted wikis) were already running. This actually caused all of the wikis to "stop()" but not "start()" again, and did not put the network changes into effect. I then decided to restart the server (using "shutdown -r now") and hopped in the car for the 15-minute drive to the datacenter. + + By the time I arrived at the datacenter, the server had come back online after a long filesystem check (due to it having been quite a while since the last one) and the various hosted wikis had come back online as well. + + Because all of the services appear to have shut down cleanly, the only likely data loss would have been the result of wiki contributors being in the middle of making an edit and then clicking "save" while the server was still down. + + Follow-up: + + We're documenting procedures such as these for future reference, and I intend to suggest a more widely-known and -supported Linux distribution for future production server deployments. + + While we are all volunteers - indeed, none of us has ever been paid for our involvement in Wiki Spot, and many of us have donated considerable sums of money to make all of this happen - it's safe to say that none of us want to operate in a mode wherein 20 minutes of unplanned downtime is acceptable. I do apologize and will strive to avoid such mishaps in the future. + + --["Users/GrahamFreeman" Graham] +
Line 27:	Line 63:
- At about 19h00 on 19 Feb 07 there were two 3-second network outages in the course of collaborative maintenance between Cernio and Cernio's vendor as we troubleshot sub-par performance issues. We solved these problems by changing the network port configurations from auto-negotiate to manually-configured full-duplex 100Mb. —["GrahamFreeman" Graham Freeman]	+ At about 19h00 on 19 Feb 07 there were two 3-second network outages in the course of collaborative maintenance between Cernio and Cernio's vendor as we troubleshot sub-par performance issues. We solved these problems by changing the network port configurations from auto-negotiate to manually-configured full-duplex 100Mb. —["Users/GrahamFreeman" Graham Freeman]
Line 43:	Line 79:
- In the early morning of Oct 22, 2006, ["GrahamFreeman"] and ["AmitVainsencher"] moved DavisWiki's server between ["Cernio"] cabinets at [http://www.sonic.net/ Sonic.net] in Santa Rosa, and [http://www.unitedlayer.com/ United Layer]'s facility at 200 Paul Ave, San Francisco. This move was in line with Cernio's overall transition to United Layer as its colocation provider. The transition was made because the new data center offered better connectivity (for example, to countries in Eastern Europe and the Middle East) coupled with nearly a 50% reduction in bandwidth costs.	+ In the early morning of Oct 22, 2006, ["Users/GrahamFreeman"] and ["Users/AmitVainsencher"] moved DavisWiki's server between ["Cernio"] cabinets at [http://www.sonic.net/ Sonic.net] in Santa Rosa, and [http://www.unitedlayer.com/ United Layer]'s facility at 200 Paul Ave, San Francisco. This move was in line with Cernio's overall transition to United Layer as its colocation provider. The transition was made because the new data center offered better connectivity (for example, to countries in Eastern Europe and the Middle East) coupled with nearly a 50% reduction in bandwidth costs.
Line 47:	Line 83:
- * Well done, gentlemen. -["JimStewart"] - * Are we big in Eastern Europe and the Middle East? -["MattJurach"] - * ''There are other sites hosted on this server, you know.'' -["WilliamLewis"] - * That, and there are plenty of other servers in the cabinet. The Cernio Tech Co-op has participants in California, New York, Hawai'i, British Columbia, England, the Czech Republic, Australia, etc... So, it's pretty important that these folks have good connectivity to the California-based servers. For Davis Wiki, it's maybe not as important, but as the world's best city wiki it's good that it remain responsive to everyone too. --["GrahamFreeman"] - Why do you think, that DavisWiki is the world's best city wiki? What makes it best? --["WilhelmBuehler"] - I'm biased. That's why it's best. :) Seriously, I mean no disrespect whatsoever to the folks who've worked on other city/community wikis - I simply enjoy this one the most because it's focused on the community I'm most familiar with. --["GrahamFreeman"]	+ * Well done, gentlemen. -["Users/JimStewart"] + * Are we big in Eastern Europe and the Middle East? -["Users/MattJurach"] + * ''There are other sites hosted on this server, you know.'' -["Users/WilliamLewis"] + * That, and there are plenty of other servers in the cabinet. The Cernio Tech Co-op has participants in California, New York, Hawai'i, British Columbia, England, the Czech Republic, Australia, etc... So, it's pretty important that these folks have good connectivity to the California-based servers. For Davis Wiki, it's maybe not as important, but as the world's best city wiki it's good that it remain responsive to everyone too. --["Users/GrahamFreeman"] + Why do you think, that DavisWiki is the world's best city wiki? What makes it best? --["Users/WilhelmBuehler"] + I'm biased. That's why it's best. :) Seriously, I mean no disrespect whatsoever to the folks who've worked on other city/community wikis - I simply enjoy this one the most because it's focused on the community I'm most familiar with. --["Users/GrahamFreeman"]

System Info

GrahamFreeman — 2007-04-04 00:46:39

Upload of image War_Room.jpg.

System Info

GrahamFreeman — 2007-04-04 00:46:23

Upload of image bandwidth05.png.

System Info

GrahamFreeman — 2007-04-04 00:46:08

Upload of image 07oct06-baxter-net-stats.png.

System Info

GrahamFreeman — 2007-04-04 00:44:21

merging with Davis Wiki system info page

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
	+ [[Image(War_Room.jpg, "The top secret wiki control center located hundreds of feet underground in a nuclear-safe bunker.*", 500, thumbnail)]] +
Line 7:	Line 9:
- ["AmitVainsencher" Amit Vainsencher] in cooperation with ["GrahamFreeman" Graham Freeman]	+ ["AmitVainsencher" Amit Vainsencher] and ["GrahamFreeman" Graham Freeman]
Line 19:	Line 21:
	+ See also the page on ["Wiki hosting"].
Line 24:	Line 27:
- At about 19h00 on 19 Feb 07 there were two 3-second network outages in the course of collaborative maintenance between Cernio and Cernio's vendor as we troubleshot sub-par performance issues. We solved these problems by changing the network port configurations from auto-negotiate to manually-configured full-duplex 100Mb. --["GrahamFreeman"]	+ At about 19h00 on 19 Feb 07 there were two 3-second network outages in the course of collaborative maintenance between Cernio and Cernio's vendor as we troubleshot sub-par performance issues. We solved these problems by changing the network port configurations from auto-negotiate to manually-configured full-duplex 100Mb. —["GrahamFreeman" Graham Freeman]
Line 32:	Line 35:
	+ == Rack re-install - November '06 == + + Just after midnight on Sunday 19 Nov '06, Amit and Graham re-installed the server in a better position in the rack. Rather than sitting on a pair of rack trays, the server is now using rackmount rails. During this process, Graham and Amit moved the server's power outlet to a managed power distribution unit (PDU), which allows authorized personnel to remotely control the power feed to the server if necessary. Depending on the circumstances, this could save a trip in to the datacenter and could therefore result in greater system uptime in the event of a problem with the server. + + Total downtime was about 20 minutes. + + == Moving Datacenters - October '06 == + + In the early morning of Oct 22, 2006, ["GrahamFreeman"] and ["AmitVainsencher"] moved DavisWiki's server between ["Cernio"] cabinets at [http://www.sonic.net/ Sonic.net] in Santa Rosa, and [http://www.unitedlayer.com/ United Layer]'s facility at 200 Paul Ave, San Francisco. This move was in line with Cernio's overall transition to United Layer as its colocation provider. The transition was made because the new data center offered better connectivity (for example, to countries in Eastern Europe and the Middle East) coupled with nearly a 50% reduction in bandwidth costs. + The transition went relatively painlessly. First, Amit made the wiki read-only, to prevent possible data loss. Then, he dumped the database and all files associated with the wiki to a system already in the new data center in SF. This was done to ensure that the site could be quickly restored if the primary server was damaged or destroyed in transit. After backing up, Graham and Amit pulled the server and loaded it into the back of Amit's car, and proceeded to drive to San Francisco (approximately 1AM). Fortunately, the backup precautions proved unnecessary, as the server was still alive and kicking on arrival. Once Graham and Amit got the server running at the new location, Graham transitioned the DNS for all of the system's domains (see above for a list) to the server's new IP address, and configured a web traffic proxy from the system's old IP in Santa Rosa to the new one in San Francisco. This was done to ensure no traffic would be lost while the new DNS information was propagating (DNS changes can take anywhere from 30 minutes to 24 hours to fully propogate). + + Once the proxy configuration was completed, Amit restored write access to the Wikis. He and Graham then left the San Francisco facility (around 4AM) in a triumphant, caffeinated haze. + * Well done, gentlemen. -["JimStewart"] + * Are we big in Eastern Europe and the Middle East? -["MattJurach"] + * ''There are other sites hosted on this server, you know.'' -["WilliamLewis"] + * That, and there are plenty of other servers in the cabinet. The Cernio Tech Co-op has participants in California, New York, Hawai'i, British Columbia, England, the Czech Republic, Australia, etc... So, it's pretty important that these folks have good connectivity to the California-based servers. For Davis Wiki, it's maybe not as important, but as the world's best city wiki it's good that it remain responsive to everyone too. --["GrahamFreeman"] + Why do you think, that DavisWiki is the world's best city wiki? What makes it best? --["WilhelmBuehler"] + I'm biased. That's why it's best. :) Seriously, I mean no disrespect whatsoever to the folks who've worked on other city/community wikis - I simply enjoy this one the most because it's focused on the community I'm most familiar with. --["GrahamFreeman"] +
Line 38:	Line 60:
	+ Here is our bandwidth usage as of early October '06: + + [[Image(07oct06-baxter-net-stats.png, noborder)]] + + Here was our bandwidth usage in 2005 at some random point: + + [[Image(bandwidth05.png, noborder)]] + + The difference from 2005 and 2006 seems to indicate we are nearly 10 times as active, at least during the day the figure was grabbed. +
Line 39:	Line 71:
	+ + You may be interested in the ["Wiki Community/Technical Discussion"]. + + ---- + * Not really. :) It's actually [wiki:WikiPedia:LIGO LIGO]

System Info

ArlenAbraham — 2007-03-05 02:43:17

+link to development

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 3:	Line 3:
- See ["Wiki Developers"]	+ See ["Development"]

System Info

GrahamFreeman — 2007-02-20 11:17:54

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
- = Who is maintaining the software that runs this site? =	+ = Who maintains the software that runs this site? =

System Info

GrahamFreeman — 2007-02-20 11:17:33

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 4:	Line 4:
	+ + = Who maintains the hardware that runs this site? = + + ["AmitVainsencher" Amit Vainsencher] in cooperation with ["GrahamFreeman" Graham Freeman]

System Info

GrahamFreeman — 2007-02-20 11:16:47

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
- = Who is maintaining the back-end of this site? =	+ = Who is maintaining the software that runs this site? =

System Info

GrahamFreeman — 2007-02-20 09:57:13

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 7:	Line 7:
- We presently operate on a single dedicated server. It has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. It is colocated in one of [http://cernio.com/ Cernio Tech Co-op]'s cabinets at United Layer's facility in the 200 Paul datacenter in San Francisco.	+ We presently operate on a single dedicated production server and two development servers - a separate physical server and a virtual private server. All three are colocated in [http://cernio.com/ Cernio Tech Co-op]'s cabinets at United Layer's facility in the 200 Paul datacenter in San Francisco, California. + + The production server has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. +

System Info

GrahamFreeman — 2007-02-20 09:02:09

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 15:	Line 15:
	+ == Network maintenance - February 2007 == + + At about 19h00 on 19 Feb 07 there were two 3-second network outages in the course of collaborative maintenance between Cernio and Cernio's vendor as we troubleshot sub-par performance issues. We solved these problems by changing the network port configurations from auto-negotiate to manually-configured full-duplex 100Mb. --["GrahamFreeman"] +

System Info

GrahamFreeman — 2007-02-20 08:56:21

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 11:	Line 11:
- See also the page on ["Wiki hosting"].	+

System Info

GrahamFreeman — 2007-02-20 08:55:38

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 7:	Line 7:
- We have our own dedicated server. It has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. We are hosted in one of [http://cernio.com/ Cernio Tech Co-op]'s cabinets at United Layer's facility in the 200 Paul data center in San Francisco.	+ We presently operate on a single dedicated server. It has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. It is colocated in one of [http://cernio.com/ Cernio Tech Co-op]'s cabinets at United Layer's facility in the 200 Paul datacenter in San Francisco. + + We have access on short notice to server capacity in London, and will soon have similar access in New York.

System Info

GrahamFreeman — 2007-02-20 06:03:54

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 7:	Line 7:
- We have our own dedicated server. It has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. We are hosted in one of [http://cernio.com/ Cernio Tech Co-op]'s cabinet at United Layer's facility in the 200 Paul data center in San Francisco.	+ We have our own dedicated server. It has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. We are hosted in one of [http://cernio.com/ Cernio Tech Co-op]'s cabinets at United Layer's facility in the 200 Paul data center in San Francisco.

System Info

GrahamFreeman — 2007-02-20 06:03:33

no image

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
- [[Image(War_Room.jpg, "The top secret wiki control center located hundreds of feet underground in a nuclear-safe bunker.*", 500, thumbnail)]] -
Line 28:	Line 26:
- - - ---- - * Is actually [wiki:WikiPedia:LIGO LIGO]

System Info

GrahamFreeman — 2007-02-20 06:01:37

(quick edit)

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 9:	Line 9:
- We have our own dedicated server. It has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. We are hosted in ["Cernio Technology Cooperative" Cernio]'s cabinet at United Layer's facility in the 200 Paul data center in San Francisco. There are several other Wikis hosted on the system, including [http://rocwiki.org/ RocWiki], [http://scwiki.org/ SCWiki], [http://ant-hill.org/ The Ant Hill Housing Cooperative], and [http://projectsycamore.org/ Project Sycamore], but DavisWiki is by far the largest.	+ We have our own dedicated server. It has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. We are hosted in one of [http://cernio.com/ Cernio Tech Co-op]'s cabinet at United Layer's facility in the 200 Paul data center in San Francisco.

System Info

BrentLaabs — 2007-02-18 02:12:25

rm old stuff

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 21:	Line 21:
- == Rack re-install - November '06 == - - Just after midnight on Sunday 19 Nov '06, Amit and Graham re-installed the server in a better position in the rack. Rather than sitting on a pair of rack trays, the server is now using rackmount rails. During this process, Graham and Amit moved the server's power outlet to a managed power distribution unit (PDU), which allows authorized personnel to remotely control the power feed to the server if necessary. Depending on the circumstances, this could save a trip in to the datacenter and could therefore result in greater system uptime in the event of a problem with the server. - - Total downtime was about 20 minutes. - - == Moving Datacenters - October '06 == - - In the early morning of Oct 22, 2006, ["GrahamFreeman"] and ["AmitVainsencher"] moved DavisWiki's server between ["Cernio"] cabinets at [http://www.sonic.net/ Sonic.net] in Santa Rosa, and [http://www.unitedlayer.com/ United Layer]'s facility at 200 Paul Ave, San Francisco. This move was in line with Cernio's overall transition to United Layer as its colocation provider. The transition was made because the new data center offered better connectivity (for example, to countries in Eastern Europe and the Middle East) coupled with nearly a 50% reduction in bandwidth costs. - The transition went relatively painlessly. First, Amit made the wiki read-only, to prevent possible data loss. Then, he dumped the database and all files associated with the wiki to a system already in the new data center in SF. This was done to ensure that the site could be quickly restored if the primary server was damaged or destroyed in transit. After backing up, Graham and Amit pulled the server and loaded it into the back of Amit's car, and proceeded to drive to San Francisco (approximately 1AM). Fortunately, the backup precautions proved unnecessary, as the server was still alive and kicking on arrival. Once Graham and Amit got the server running at the new location, Graham transitioned the DNS for all of the system's domains (see above for a list) to the server's new IP address, and configured a web traffic proxy from the system's old IP in Santa Rosa to the new one in San Francisco. This was done to ensure no traffic would be lost while the new DNS information was propagating (DNS changes can take anywhere from 30 minutes to 24 hours to fully propogate). - - Once the proxy configuration was completed, Amit restored write access to the Wikis. He and Graham then left the San Francisco facility (around 4AM) in a triumphant, caffeinated haze. - * Well done, gentlemen. -["JimStewart"] - * Are we big in Eastern Europe and the Middle East? -["MattJurach"] - * ''There are other sites hosted on this server, you know.'' -["WilliamLewis"] - * That, and there are plenty of other servers in the cabinet. The Cernio Tech Co-op has participants in California, New York, Hawai'i, British Columbia, England, the Czech Republic, Australia, etc... So, it's pretty important that these folks have good connectivity to the California-based servers. For Davis Wiki, it's maybe not as important, but as the world's best city wiki it's good that it remain responsive to everyone too. --["GrahamFreeman"] - Why do you think, that DavisWiki is the world's best city wiki? What makes it best? --["WilhelmBuehler"] - I'm biased. That's why it's best. :) Seriously, I mean no disrespect whatsoever to the folks who've worked on other city/community wikis - I simply enjoy this one the most because it's focused on the community I'm most familiar with. --["GrahamFreeman"] -

System Info

BrentLaabs — 2007-02-18 01:53:21

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 46:	Line 46:
- Here is our bandwidth usage as of early October '06: - - [[Image(07oct06-baxter-net-stats.png, noborder)]] - - Here was our bandwidth usage in 2005 at some random point: - - [[Image(bandwidth05.png, noborder)]] - - The difference from 2005 and 2006 seems to indicate we are nearly 10 times as active, at least during the day the figure was grabbed. -

System Info

BrentLaabs — 2007-02-18 01:49:26

stolen from davis wiki

Differences for System Info

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
	+ [[Image(War_Room.jpg, "The top secret wiki control center located hundreds of feet underground in a nuclear-safe bunker.", 500, thumbnail)]] + + = Who is maintaining the back-end of this site? = + + See ["Wiki Developers"] + + = What is your server setup? = + + We have our own dedicated server. It has a 2.0GHz Athlon64 dual core processor (an x2 3200+) with 4GB of ECC memory and 200GB of RAID1 storage. The server is based on a [http://www.tyan.com/products/html/gt20b2865.html Tyan Transport] barebone package and runs [http://www.gentoo.org/ Gentoo Linux]. We are hosted in ["Cernio Technology Cooperative" Cernio]'s cabinet at United Layer's facility in the 200 Paul data center in San Francisco. There are several other Wikis hosted on the system, including [http://rocwiki.org/ RocWiki], [http://scwiki.org/ SCWiki], [http://ant-hill.org/ The Ant Hill Housing Cooperative], and [http://projectsycamore.org/ Project Sycamore], but DavisWiki is by far the largest. + + See also the page on ["Wiki hosting"]. + + = Recent Hardware Maintenance = + + == Network hardware change - December '06 == + + Shortly before midnight on Thursday 7 Dec '06, Graham moved the server's network connection from one network switch to another. This resulted in two periods of downtime of approximately 15 seconds each. This was done in preparation for making significant changes to the configuration on the first switch. When that's done, we'll move back to the first one and make similar changes to the second switch. The end result will be better network management, greater expansion capabilities, and better protection against hardware failures. + + Total downtime was about 30 seconds. + + == Rack re-install - November '06 == + + Just after midnight on Sunday 19 Nov '06, Amit and Graham re-installed the server in a better position in the rack. Rather than sitting on a pair of rack trays, the server is now using rackmount rails. During this process, Graham and Amit moved the server's power outlet to a managed power distribution unit (PDU), which allows authorized personnel to remotely control the power feed to the server if necessary. Depending on the circumstances, this could save a trip in to the datacenter and could therefore result in greater system uptime in the event of a problem with the server. + + Total downtime was about 20 minutes. + + == Moving Datacenters - October '06 == + + In the early morning of Oct 22, 2006, ["GrahamFreeman"] and ["AmitVainsencher"] moved DavisWiki's server between ["Cernio"] cabinets at [http://www.sonic.net/ Sonic.net] in Santa Rosa, and [http://www.unitedlayer.com/ United Layer]'s facility at 200 Paul Ave, San Francisco. This move was in line with Cernio's overall transition to United Layer as its colocation provider. The transition was made because the new data center offered better connectivity (for example, to countries in Eastern Europe and the Middle East) coupled with nearly a 50% reduction in bandwidth costs. + The transition went relatively painlessly. First, Amit made the wiki read-only, to prevent possible data loss. Then, he dumped the database and all files associated with the wiki to a system already in the new data center in SF. This was done to ensure that the site could be quickly restored if the primary server was damaged or destroyed in transit. After backing up, Graham and Amit pulled the server and loaded it into the back of Amit's car, and proceeded to drive to San Francisco (approximately 1AM). Fortunately, the backup precautions proved unnecessary, as the server was still alive and kicking on arrival. Once Graham and Amit got the server running at the new location, Graham transitioned the DNS for all of the system's domains (see above for a list) to the server's new IP address, and configured a web traffic proxy from the system's old IP in Santa Rosa to the new one in San Francisco. This was done to ensure no traffic would be lost while the new DNS information was propagating (DNS changes can take anywhere from 30 minutes to 24 hours to fully propogate). + + Once the proxy configuration was completed, Amit restored write access to the Wikis. He and Graham then left the San Francisco facility (around 4AM) in a triumphant, caffeinated haze. + Well done, gentlemen. -["JimStewart"] + * Are we big in Eastern Europe and the Middle East? -["MattJurach"] + * ''There are other sites hosted on this server, you know.'' -["WilliamLewis"] + * That, and there are plenty of other servers in the cabinet. The Cernio Tech Co-op has participants in California, New York, Hawai'i, British Columbia, England, the Czech Republic, Australia, etc... So, it's pretty important that these folks have good connectivity to the California-based servers. For Davis Wiki, it's maybe not as important, but as the world's best city wiki it's good that it remain responsive to everyone too. --["GrahamFreeman"] + Why do you think, that DavisWiki is the world's best city wiki? What makes it best? --["WilhelmBuehler"] + I'm biased. That's why it's best. :) Seriously, I mean no disrespect whatsoever to the folks who've worked on other city/community wikis - I simply enjoy this one the most because it's focused on the community I'm most familiar with. --["GrahamFreeman"] + + = What are some statistics on the site? = + + There are '''[[PageCount]]''' pages and '''[[UserCount]]''' registered users on this wiki. + + You can find some auto-generated stats [http://baxter.cernio.com/mrtg/ here]. They are updated every 5 minutes or so. + + Here is our bandwidth usage as of early October '06: + + [[Image(07oct06-baxter-net-stats.png, noborder)]] + + Here was our bandwidth usage in 2005 at some random point: + + [[Image(bandwidth05.png, noborder)]] + + The difference from 2005 and 2006 seems to indicate we are nearly 10 times as active, at least during the day the figure was grabbed. + + We also have lots of individual ["User Statistics"]. + + + ---- + * Is actually [wiki:WikiPedia:LIGO LIGO]