Troubleshooting

<slideshow style="nobleprog" headingmark="⌘" incmark="…" scaled="true" font="Trebuchet MS" >

title

Troubleshooting

author

Bernard Szlachta (NobleProg Ltd)

</slideshow>

Troubleshooting ⌘

• Before you assume it's an nginx problem, take a step back and look at the issue
• Automated script to help problem diagnosis - What's Wrong?

• https://github.com/BashtonLtd/whatswrong

nginx Troubleshooting ⌘

• Always syntax check your config

service nginx configtest

• Check the error log

• By default, /var/log/nginx/error_log

nginx error logs ⌘

• Can sometimes be a little cryptic
• Google helps of course..

Common errors (1) ⌘

2012/11/29 06:31:42 [error] 2589#0: *6437 client intended to send too 
large body: 13106010 bytes, client: 127.0.0.1, server: , request: 
"POST /upload/ HTTP/1.1", host: "www.example.com", referrer: 
"http://www.example.com/file_upload/"

• Client attempted to upload a file bigger than the maximum POST size

Common errors (1) ⌘

2012/11/29 06:31:42 [error] 2589#0: *6437 client intended to send too 
large body: 13106010 bytes, client: 127.0.0.1, server: , request: 
"POST /upload/ HTTP/1.1", host: "www.example.com", referrer: 
"http://www.example.com/file_upload/"

Common errors (2) ⌘

2012/10/14 19:51:22 [emerg] 3969#0: "proxy_pass" cannot have 
URI part in location given by regular expression, or inside 
named location, or inside "if" statement, or inside 
"limit_except" block in /etc/nginx/conf.d/proxy.conf:16

Common errors (2)⌘

2012/10/14 19:51:22 [emerg] 3969#0: "proxy_pass" cannot have 
URI part in location given by regular expression, or inside 
named location, or inside "if" statement, or inside 
"limit_except" block in /etc/nginx/conf.d/proxy.conf:16

• nginx won't start to to configuration syntax error
• Line containing error shown
• If you attempted to reload nginx with this config, it will continue to run with the old config

Common errors (3) ⌘

2012/10/14 18:46:54 [emerg] 2593#0: unknown directive 
"client_body_temp_path" in /etc/nginx/conf.d/vhost.conf:6

Common errors (3) ⌘

2012/10/14 18:46:54 [emerg] 2593#0: unknown directive 
"client_body_temp_path" in /etc/nginx/conf.d/vhost.conf:6

• client_body_temp_path is actually a valid directive
• .. but it's being used in the wrong place
• Because nginx is modular, each module's directives are known only inside the section that module configures

Common errors (4) ⌘

2012/10/16 20:56:31 [emerg] 3039#0: "try_files" directive is not allowed
here in /opt/nginx/conf/vhost.conf:16

Common errors (4) ⌘

2012/10/16 20:56:31 [emerg] 3039#0: "try_files" directive is not allowed
here in /etc/nginx/conf.d/vhost.conf:16

• nginx can sometimes hint when you've used something valid for the module, but in the wrong place

Common errors (5) ⌘

2012/10/16 20:56:42 [emerg] 3043#0: host not found in upstream "tickets.
example.com" in /opt/nginx/conf.d/proxy.conf:22

Common errors (5) ⌘

2012/10/16 20:56:42 [emerg] 3043#0: host not found in upstream "tickets.
example.com" in /opt/nginx/conf.d/proxy.conf:22

• Your configuration is valid, but there is a DNS problem looking up

 tickets.example.com

Common errors (6) ⌘

2012/10/29 18:59:26 [emerg] 2287#0: unexpected "}" in /etc/nginx/conf.d/
vhost.conf:40

Common errors (6) ⌘

2012/10/29 18:59:26 [emerg] 2287#0: unexpected "}" in /etc/nginx/conf.d/
vhost.conf:40

• Simple syntax error
• Too many closing brackets

Common errors (7)

2012/10/29 18:59:26 [emerg] 2287#0: unexpected end of file, expecting "}" 
in /etc/nginx/conf.d/ vhost.conf:40

Common errors (7) ⌘

2012/10/29 18:59:26 [emerg] 2287#0: unexpected end of file, expecting "}" 
in /etc/nginx/conf.d/ vhost.conf:40

• Simple syntax error
• Insufficient closing brackets

Debugging with custom logs ⌘

• Not sure if a section of config is getting called, or what a variable is set to?
• Log data into the access logs with the `log_format` directive

Debugging with custom logs example ⌘

http {
 log_format sentlog '[$time_local] "$request" $status $body_bytes_sent ';
 log_format imagelog '[$time_local] $image_file $image_type ' '$body_bytes_sent $status';
 log_format authlog '[$time_local] $remote_addr $remote_user ' '"$request" $status';
 server {
   server_name .example.com;
   root /home/www;
   location / {
     access_log logs/example.com-access.log combined;
     access_log logs/example.com-root_access.log sentlog;
     rewrite ^/(.*)\.(png|jpg|gif)$ /images/$1.$2;
     set $image_file $1;
     set $image_type $2;
   }
   location /images {
     access_log logs/example.com-images_access.log imagelog;
   }
   location /auth {
     auth_basic "authorized area";
     auth_basic_user_file conf/htpasswd;
     deny all;
     access_log logs/example.com-auth_access.log authlog;
   }
 }
}

Miscellaneous ⌘

Be careful what you believe on the Internet ⌘

• nginx still relatively young
• There have been many changes to configuration syntax and features since release
• Many blog/forum posts you will find via Google telling you how to do something are outdated or just plain wrong

Common mis-use ⌘

• Commonly seen on forums/blogs:

location / {
 if (!-f $request_filename) {
   include fastcgi_params;
   proxy_pass 127.0.0.1:8000;
   break;
 }
}

Common mis-use - the right way ⌘

• A better performing way to achieve the same result:

location / {
 try_files $uri $uri/ @proxy;
}

location @proxy {
 proxy_pass 127.0.0.1:8000;
}

More commonly seen bad advice ⌘

server {
 server_name .example.com;
 root /var/www/html;
 if ($host ~* ^example\.com) {
   rewrite ^/(.*)$ http://www.example.com/$1 redirect;
 }
}

The right way ⌘

server {
 server_name example.com;
 return 301 http://www.example.com;
}

In-place upgrades ⌘

• nginx has a nice feature to switch binaries while running
• installing an upgraded package leaves the old version of nginx running
• simply run `service nginx upgrade` to switch to the new version with 0 downtime

nginx status module ⌘

• nginx has module which can provide status details vi a URL

location /nginx_status {
 stub_status on;
 access_log off;
 allow 127.0.0.1;
 deny all;
}

Status module output ⌘

$ curl -s http://localhost/nginx_status
Active connections: 2532
server accepts handled requests
1476737983 1476737983 3553635810
Reading: 93 Writing: 13 Waiting: 2426