Thursday, February 4, 2010

Squid + Apt-proxy: transparent proxy

There really isn't much information on how to get Squid and AptProxy to work together, so since I've finally got around to polishing it up, I figured I'd share.

To set up transparent proxy to your Apt-Proxy Cache, you first need to set up a transparent proxy with Squid.

Then you need a url_rewrite_program (see the squid.conf file). For our lab set-up I decided to use squidGuard, since it seemed to be designed for squid. There are rumors that a simple perl script could handle the rewriting, and for this project that would be enough.

However, I'm going to stick to squidGuard for this. It'll give a good idea how to do it in Perl in anycase.
In your squidGuard file, add the code:
rew aptproxy {
    s@^http://ke.archive.ubuntu.com/ubuntu/@http://10.42.43.1:9999/ubuntu/@
    s@^http://archive.ubuntu.com/ubuntu/@http://10.42.43.1:9999/ubuntu/@
    s@^http://archive.canonical.com/ubuntu/@http://10.42.43.1:9999/partner/@
    s@^http://security.ubuntu.com/ubuntu/@http://10.42.43.1:9999/ubuntu-security/@
}

This is with the apt-proxy config:
[ubuntu]
backends =
    http://ke.archive.ubuntu.com/ubuntu
    http://archive.ubuntu.com/ubuntu

[ubuntu-security]
backends = http://security.ubuntu.com/ubuntu

[partner]
backends = http://archive.canonical.com/ubuntu
 

Then make sure you call it in your 'acl'
acl {
    default {
        pass    !bad all
        rewrite     aptproxy
        redirect    http://www.domain.com/blocked.html
   }
}

This re-writes Ubuntu repositories to point to the local apt-proxy cache. And your job may be done.
For me, however, I had alot of trouble with corruption. Sometimes a file would be corrupt and since the url "ke.archive.ubuntu.com" is caught by squid first, squid caches it. Then both squid and apt-proxy would have a corrupt file. (in this case a *.bz2 file)

So we need to set up squid to not cache these requests:
acl localnet src 10.42.43.0/255.255.255.0
acl ubuntu_repo dstdomain .archive.ubuntu.com security.ubuntu.com archive.canonical.com
cache deny ubuntu_repo
cache deny localnet
cache allow all
(the dot infront of archive.ubuntu.com catches sub-domains)
There may be other ways to go about doing this. I'm interested in squid's 'cache_peer' configuration but I'm unsure if it works since the urls need to be re-written for Apt-proxy.

Tuesday, January 5, 2010

Eee PC with Ubuntu (update)

I've done a bit of research today, and got a bit of a better "battery saving" setup for my laptop.

Saving your SSD disk

The default settings for Ubuntu are built to work with a spinning disk. Since you really want your disk to last as long as possible, it's better to minimize the disk writes.
Edit your fstab so that specific directories are sent to memory instead of disk:
tmpfs      /var/log        tmpfs        defaults           0    0
tmpfs /tmp tmpfs defaults 0 0
tmpfs /var/tmp tmpfs defaults 0 0
tmpfs /var/log/apt tmpfs defaults 0 0
From help.ubuntu.com.
The one thing that is annoying though, is anytime you run aptitude, it throws an error about /var/log/apt which it can't make...
So I added "mkdir /var/log/apt" to a start script.

Then You can point different applications to /tmp, for example Firefox:

  • Go to the address 'about:config.'
  • Create a new value 'browser.cache.disk.parent_directory' as a string and set it to '/tmp'
At first I was afraid it would just dump everything there, but don't worry. It makes the folder "Cache" in /tmp

Other Hard Drive tweaks

Another minor fix with the disk is to switch to "deadline". Run:
cat /sys/block/sda/queue/scheduler
and you'll probably get something like:
noop anticipatory deadline [cfq]
Apparently deadline and noop are the options, but deadline is better. Set it to deadline with this:
echo deadline > /sys/block/sda/queue/scheduler
This will need to be a start script.

We can also add FIFO queuing by running the command:

echo 1 > /sys/block/sda/queue/iosched/fifo_batch
Again, this resets on boot, so you'll need to make a start script.

Disabling/Controlling Hardware features

First off, You should install eee-control. The version for Karmic just recently came out, and isn't quite the same now that the developer is declared missing, but it will still turn on and off most of the things you need.

The one thing to note, is that I still haven't figured out how to get wireless to work. It's easy to turn on, but crashes when you turn it off. However, if you turn it off on start up (ie before the gui loads), then somehow it doesn't crash.

In anycase, I like to turn everything off on startup, so I'm going to add the following to a start script:

# camera
echo 0 > /sys/devices/platform/eeepc/camera
# card reader
echo 0 > /sys/devices/platform/eeepc/cardr
# wireless
echo 0 > /sys/devices/platform/eeepc/rfkill/rfkill0/state
That way if I need it, I'll turn it on. If I don't, then it won't use up battery. And Eee-control-tray makes it very easy to enable features when I need them.

Power Management

Ubuntu has some built in things to extend laptop battery life, and still maintain full power while plugged in, but not nearly what it could be.
For example. What if we want to turn our computer's proc the whole way up while we're on AC, turn brightness the whole way up, increase disk writes, and have the screen dim after 10 minutes:
echo 0 > /proc/sys/vm/laptop_mode
# turn brightness up the whole way
echo 15 > /sys/devices/platform/eeepc/backlight/eeepc/brightness
echo 10 > /proc/sys/vm/dirty_ratio
echo 5 > /proc/sys/vm/dirty_background_ratio
echo 500 > /proc/sys/vm/dirty_writeback_centisecs
# screen off after 10 minutes
xset dpms 0 0 600
Well, then we want quite a bit different for when the laptop goes into battery mode.
Check this out:
#!/bin/bash
if on_ac_power; then
# Reset back to normal settings
echo 0 > /proc/sys/vm/laptop_mode
# turn brightness up the whole way
echo 15 > /sys/devices/platform/eeepc/backlight/eeepc/brightness
# disk
echo 10 > /proc/sys/vm/dirty_ratio
echo 5 > /proc/sys/vm/dirty_background_ratio
echo 500 > /proc/sys/vm/dirty_writeback_centisecs
# proccessor on performance
echo 0 > /sys/devices/platform/eeepc/cpufv
# screen off after 10 minutes
xset dpms 0 0 600
else
# Turn on aggressive power savings
echo 5 > /proc/sys/vm/laptop_mode
# turn brightness down the whole way
echo 0 > /sys/devices/platform/eeepc/backlight/eeepc/brightness
# disk
echo 40 > /proc/sys/vm/dirty_ratio
echo 10 > /proc/sys/vm/dirty_background_ratio
echo 1500 > /proc/sys/vm/dirty_writeback_centisecs
# proccessor on powersave
echo 2 > /sys/devices/platform/eeepc/cpufv
# screen off after 2 minutes
xset dpms 0 0 120
fi
It just so happens that "on_ac_power" returns true if it's on AC power, and false if it's not.

This should be a power script (/usr/lib/pm-utils/power.d/).

Note: on a clean install this should work, though there have been some issues. Make sure you don't link, but actually copy the script, and have the right permissions. Using the "install" tool is recommended.

Start scripts (and action scripts)

So since this post depends so much on action scripts, here's how to make them.

The folders for the ones we've found today are as follows:

 scripts run on startup:    /etc/rcS.d/
scripts on power (on/off): /usr/lib/pm-utils/power.d
The job is pretty easy. What you need to do is write the script, copy it to the folder, and then set it to be executable (the last two actions can be done with one command: the "install" tool.

First, write the script. In this case I have three start scripts:

S95make_tmp_folders

 # apt log
mkdir /var/log/apt\

S96disk_editing

 # Deadline
echo deadline > /sys/block/sda/queue/scheduler
# FIFO
echo 1 > /sys/block/sda/queue/iosched/fifo_batch

S97turn_off_hardware

 # camera
echo 0 > /sys/devices/platform/eeepc/camera
# card reader
echo 0 > /sys/devices/platform/eeepc/cardr

After each file is written, simple run:

install {filename} /etc/rcS.d/
and the file will be installed to this folder.

In the case of our power script we want it to be run on start up, power flux. So we'll run:


install 99power_save /usr/lib/pm-utils/power.d
install 99power_save /etc/rcS.d
Where 99power_save is the name of the script that we wrote above.

Wednesday, December 16, 2009

Tale of Three Music Players

Once upon a time I used Amarok. Those were the good days. MySql Databases with statistics, etc. Then Amarok 2 came along. I figured out a way to go back to my former love, with the capabilities instead of cheesy looks.
But this didn't last forever, for then I got a Eee PC, and due to my laziness, I didn't install amarok1.4 (the good one). Instead, since I had lost my database, I tested Rhythmbox.

Rhythmbox was good to me for a while. We played many a piece of music together. Then a friend encouraged me to test out Banshee. Banshee did scores just like Amarok did! So I tried it out.

Banshee had everything I would want in a music player. It had scores (though not nearly as good as I had remembered amarok), it handled libraries for videos, and even though I couldn't embed my tags in the videos, it tagged the videos. What Rhythmbox didn't have was a good library organization, and Banshee handled that too! (kinda... it sorta did it while it was going instead of all at once like I would have wanted... leaving half my music sorted and the other half messy in the same folder).

But Banshee was still new, and so didn't have a good relationship with mp3s. In fact, my poor flash drive was being read/written the whole time the song was playing, so I decided that this was reason enough to go back to Rhythmbox. After all, it's the tried and true that keep us going. Besides, Rhythmbox could handle me dropping new files into the folder right away.

Soon, I bought a CD. Without proper CD ripping equipment, I was forced to copy the *.wav files off the CD, and then encode to mp3. But the mp3s were tagless.
So I tagged them in Rhythmbox.
But Rhythmbox couldn't sort them. Instead I had to deal with "Track 1.mp3" files. Which was unacceptable. If I do this with another CD, the files would get replaced!

Then I discovered a program. EasyTag. EasyTag easily tags the mp3s and sorts them.

So I ran it. The mp3s was now organized.

But my playcounts!! They were gone! Rhythmbox couldn't handle the files being in a new location!


So then I remembered my former love. Even without MySql Amarok is superior in all functionality.


Amarok:
  • Score and Playcounts (which happen to be smarter then Banshee)

  • Scripts/Plugins

  • Lyric support (you know... for my Swahili/Kikuyu music)

  • Last.fm Support (+ queuing if offline)

  • Smart Playlists

  • Good Random tool for Playlist

  • Organize Mode

  • Album Cover Organizer

  • (+ A lot of other options.. in 1.4)


Rhythmbox:
  • Playcount

  • Hot Scanning

  • Very stable

  • Random tool for Now Playing

  • Lyric searching

  • Last.fm support (+ Queuing if offline)

  • Easy to add covers to CURRENTLY playing song


Banshee:
  • Playcount/Score (inferior)

  • Lyric support

  • Unstable

  • Last.fm

  • No organize mode, but it organizes...kinda

  • Video Library!

Tuesday, November 24, 2009

Eee PC (Ubuntu)

The Eee PC is quite a effective, and small unit. It's lasted three months in my hands, which is probably a good sign (realizing how I've screwed up the keyboard with soapy-wet hands)

Ubuntu Netbook remix made a big leap in the latest release, also.
Maximus seems more stable, and the layout is quite a bit better.



The main problem is when you switch to Ubuntu there are certain features, like disabling:
- Camera
- Card Reader
- Wireless
That you can't fully control. Sure you can disable the "wireless" in Ubuntu, but the actual wireless card is still on unless you disable it in bios.

eee-control fixes this though.
The version in Jaunty is quite effective. It controls wireless, camera, card reader, brightness, processor and fan speeds, and probably some other things I forgot.

If you like the newest version of Ubuntu (9.10), then you're out of luck. Building eee-control fails.
Rumor has it that the karmic build will be available soon.

Wednesday, November 18, 2009

Mutt: Scripting mail

It'll be helpful to ssh into the computers, in case there's an easy fix that doesn't require traveling across Kenya to fix.
ssh is quite a bit better then vnc to try and use, but in both cases, the IP address is needed.

The IP address changes on each IP up though.
So basically we need to know the IP after it changes, assuming that the user has a problem.
So, what we could do is ask the user to send us the IP over mobile, when they call, or we could have the computer itself send an e-mail.

With 'mutt' we can generate e-mails and have them sent to our default e-mail address, in this case me@domain.com
mutt -s "Subject line here" me@domain < /path/to/message
is the correct command for this e-mail address. You can find more details at http://www.cyberciti.biz/tips/sending-mail-with-attachment.html or the man page of mutt.

In the end, my script looks like this:
# Sends the IP address to default address
to="me"
from="noreply"
domain="domain.com"
temp=`mktemp -t`
ifconfig ppp0 > $temp
mutt -e "set from=$from@$domain" -s "IP address - $HOSTNAME" "$to@$domain" < $temp
with the assumption that $HOSTNAME has important information about the location of the computer (ie: name of school, etc). The main addition is the "-e" command.
If you look online you'll find that in order to send from a specific domain, you need to set up a muttrc file. Since we're running a script, and it makes more sense to keep everything together, I decided it's better to just add the "set from=" command in the script instead of an outside file.
The "-e" command lets you do this. It must be in quotes, otherwise it'll throw an error.

This is also added to the ip-up script that is run when ppp0 makes a connection.

Reloading PPP0 for a stable network

So the organization of the lab at this point is such that one computer is plugged into the modem, and the rest of the computers use this computer as a router. They are all connected to each other through a switch. You can see my project plan for more details.

It turns out, that a few things need to be done before the connection works one hundred percent like it should. Maybe it's because the computer needs to be made aware of the ppp0 connection each time it's created.

Note: the ppp0 connection is created after the user logs in. If it was created much earlier, alot of this wouldn't matter. The ppp0 connection is also very unstable and may disconnect a few times in between.

In anycase, there are 4 things that should happen:

1) Squid needs to be reconfigured for the ppp0 dns servers. If you research squid, you'll know that you can easily add the dns servers into the configuration file. However, I decided that this way was better, incase something should happen. While DNS servers probably never change, resetting it for the /etc/resolv.conf file (containing the dns addresses, and automatically created when ppp0 goes up) works just as well.
2) iptables needs to be recreated. While this solution is not very elegant, it was required to do the trick. I'll refrain from making up a solution, because it'll only show my ignorance.
3) eth0 has to be restarted. The eth0 connection was set to "share with other computers" but it doesn't share the ppp0 connection on boot up because the ppp0 connection is created later.
4) Apt-proxy needs to be restarted. Along with the restart, I figured we might as well bunzip all the bz2 files that will cause apt-get to hang.
5) An e-mail should be sent to the e-mail address, allowing me to ssh into the computer later.

The script below was then saved in /etc/ppp/ip-up.d/
where all scripts within are run when the connection is made.

So following is my script.
#!/bin/bash

# This is a list of commands that need to be run in order for squid, and apt-proxy to perform properly.
# The list is a bit ridiculous... hopefully it doesn"t take too long.
# Run the commands in order of urgency

### SQUID ###
# Resets Squid for the new DNS servers. This is the best solution incase the provider changes DNS servers
# The other option is to copy the dns servers into squid.conf
squid3 -k reconfigure

### IPTABLES ###
# Rebuilds IP tables...
# Probably shouldn"t need to be the fix, but i iz a newb.
# Plus it works.... and doesn"t take much time at all...
s_ip="10.42.43.1"
inet="ppp0"
lan="eth0"
s_port="3128"
# Clear everything
iptables -F; iptables -X; iptables -t nat -F; iptables -t nat -X; iptables -t mangle -F; iptables -t mangle -X; iptables -t raw -F; iptables -t raw -X
# Build
iptables -t nat -A PREROUTING -i $lan -p tcp -m tcp --dport 80 -j DNAT --to-destination $s_ip:$s_port
iptables -t nat -A PREROUTING -i $inet -p tcp -m tcp --dport 80 -j REDIRECT --to-ports $s_port


### ETH0 ###
# Restarts the eth0 connection, which allows it to share the new ppp0 connection over the eth0 connection
ifconfig eth0 down
ifconfig eth0 up


### E-MAIL ADDRESS ###
# Sends the IP address by e-mail
# For the e-mail code, see the email post.

### APT-PROXY ###
# bunzip apt-proxy lists. a bug in apt-proxy that causes it to hang
# NOTE: should be run last. It"s the slowest.
find /var/cache/apt-proxy/ -name *.bz2 -exec bunzip2 -f {} \;
$printing "$this: the apt-proxy cache has been bunzip2'd"
# Restart Apt-proxy. Needs to be done after bunzip2
/etc/init.d/apt-proxy restart

Squid -- Transparency

A few guides gave ideas on how to get iptables to work:
# squid server IP
SQUID_SERVER="10.42.43.1"
# Interface connected to Internet
INTERNET="ppp0"
# Interface connected to LAN
LAN_IN="eth0"
# Squid port
SQUID_PORT="3128"
# Add to IP tables
iptables -t nat -A PREROUTING -i $LAN_IN -p tcp -m tcp --dport 80 -j DNAT --to-destination $SQUID_SERVER:$SQUID_PORT
iptables -t nat -A PREROUTING -i $INTERNET -p tcp -m tcp --dport 80 -j REDIRECT --to-ports $SQUID_PORT
I added my own code to a basically do a full flush before I added to the table:
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t raw -F
iptables -t raw -X
clearly more destructive then it needs to be, but it clears everything.

The problem is on restarting the computer, it seemed to disappear. So I thought I had figured it out, until it didn't work anymore.
Evenutally I figured out I needed to edit "/etc/sysctl.conf"
28c28
< #net.ipv4.ip_forward=1 --- > net.ipv4.ip_forward=1


Unfortunately this still didn't fix the problem. The packets were forwarded, but not through squid.
Eventually I ended up re-building it each time ppp0 was created by putting it as a process in ip-up.d/

Oh, also don't forget to add "transparent" to the line in the squid.conf file.
http_port 3128 transparent


To learn more about iptables: http://www.frozentux.net/documents/iptables-tutorial/