Commit Graph

446 Commits

Author SHA1 Message Date
Seth Jennings
fd9c6d2fde adaptive longOp for du operation 2017-10-05 09:22:54 -05:00
David Ashpole
888a529088 fix #1743; move off of docker/engine-api 2017-09-28 11:05:13 -07:00
David Ashpole
e6b6a1ac57 fix #1708; move from inotify to fsnotify 2017-09-28 10:57:49 -07:00
Antonio Murdaca
4b002b3bd3
*: add CRI-O handler
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2017-09-05 17:01:58 +02:00
Euan Kemp
d2e11efba2 libcontainer: use real number of CPUs for usage
As of the 4.7 kernel, the cpustats field returned from libcontainer
contains values for every possible cpu (including nonexistent ones).
The extra values are all 0s.

If we assume that hotplug events won't happen, we can get a more
accurage cpu count by using runtime.NumCPU and then ignoring any values
beyond that.
2017-08-30 14:26:26 -07:00
Karl
5a033c064e Add runtime options for TLS support 2017-08-24 15:14:33 +02:00
Derek Carr
6fa48d9048 Expose total_rss when hierarchy is enabled 2017-08-23 14:56:59 -04:00
Derek Carr
d493f11f0b Reduce log spam when unable to get network stats 2017-08-18 16:11:03 -04:00
Andrew Pilloud
9e40f1c837 Fix race in ignoring non-existent files 2017-06-08 11:16:31 -07:00
David Ashpole
96b603e20a Merge branch 'master' into overlay2 2017-05-31 14:05:54 -07:00
David Ashpole
de308101e4 Merge branch 'master' into container-restart-count 2017-05-30 11:33:39 -07:00
Ricardo Pchevuzinske Katz
f5a896ef4c Gather information from RestartCount counter and put into labels 2017-05-30 14:05:14 -03:00
Andy Goldstein
69c085d9fa Cache the Docker thin pool name
Cache the Docker thin pool name in the dockerFactory and pass it to each Docker container handler,
instead of calling `docker info` each time a new container handler is created, as the thin pool name
is extremely unlikely to change.
2017-05-30 11:50:58 -04:00
Clayton Coleman
4e25a7951f
Report container FS metrics into prometheus /metrics
PerDiskStats reported from cgroups were not being surfaced into
prometheus. In order to properly correlate the metrics, we need to
assign a device label to each metric (which is the FS or device path).
Since blkio cgroup tracks devices, we create a synthetic device
`/dev/NAME` for the metric.

Assign a Device label to each PerDiskStat for the handlers up front, and
then surface the PerDiskStat values into the prometheus metrics. Report
two new metrics - total bytes read and total bytes written.
2017-04-24 20:46:55 -04:00
Bilal Amarni
e0c8ec569d [docker] add overlay2 storage driver
Signed-off-by: Bilal Amarni <bilal.amarni@gmail.com>
2017-04-13 13:39:45 +02:00
Tristan Colgate
227bb3a79d Add udp and udp6 network statistics 2017-04-10 20:41:51 +01:00
Manjunath A Kumatagi
8fb1158353 Add Docker API version 2017-04-04 10:56:11 +05:30
Brian Akins
feecd47daa Add watcher for zfs similar to devicemapper
minor cleanup

ensure we look at parent dataset for limit, etc
2017-03-15 18:31:11 -04:00
Derek Carr
b172ba65b4 disable thin_ls due to excessive iops 2017-02-09 10:29:58 -05:00
Calum Lacroix
6d67091c8b Checks for successful string split on env variable 2017-02-08 22:39:34 +00:00
Calum Lacroix
967e09a3a5 Checks environment variables for empty strings
If an environment variable is an empty string, cadvisor panics and fails to start
2017-02-08 18:44:56 +00:00
Derek Carr
4ad5c58f6a Ignore any .mount cgroup in docker handler 2017-01-16 11:02:37 -05:00
David Ashpole
9fdeefe3e4 Cadvisor now publishes per-container inode stats using the command 'find . -xdev printf '.'| wc -c' this is published in the v2 api using a new field 2016-10-12 13:16:21 -07:00
Tim St. Clair
afe67fec68
Cleanup comment style: // should be proceded by a space 2016-10-07 17:06:02 -07:00
derekwaynecarr
b84046f12c Look at all cgroup mounts 2016-09-22 15:34:59 -04:00
Chun Chen
844c98e073 GetSpec missing memory reservation 2016-08-16 23:26:46 +08:00
Florian Koch
3ce98a46c4 add cgropu swap usage and export as prometheus metric 2016-08-09 07:33:37 +02:00
Tobias Schmidt
1653733ea7 Expose cpu cgroup CFS prometheus metrics
If CPU quota is configured (cpu.cfs_quota != -1) the CFS will provide
stats about elapsed periods and throtting in cpu.stats. This change
makes these information available as container_cpu_cfs_* metrics.
2016-08-06 18:08:26 -04:00
Andy Goldstein
2b525ff87e Ensure minimum kernel version for thin_ls
Ensure that kernel >= 4.4.0 or RHEL/Centos 7 kernel >= 3.10.0-366 exists before starting the thin
pool watcher. Prior versions have a bug in which reserving and releasing the metadata snapshot can
cause thin pool corruption.
2016-08-04 16:14:40 -04:00
derekwaynecarr
6c114be580 Expose total inodes 2016-08-02 10:47:51 -04:00
Tim St. Clair
bbe54c0a10 Merge pull request #1368 from derekwaynecarr/inodes_pointer
Allow clients to know if inodes data is available
2016-07-29 10:10:30 -07:00
derekwaynecarr
cccf9d5fec Allow clients to know if inodes are supported on a filesystem 2016-07-26 11:15:07 -04:00
Andy Goldstein
d05098b314 Lower ThinPoolWatcher cache miss log level 2016-07-23 07:07:42 -04:00
Michael Taufen
307d1b1cb3 Modify working set memory stats calculation
Change working set calculation to usage - total_inactive_file, rather than
usage - total_inactive_anon - total_inactive_file. Since writes to tmpfs
get tracked as total_inactive_anon when swap is disabled, the old
calculation would under-report memory pressure.

See this Kubernetes issue for context:
https://github.com/kubernetes/kubernetes/issues/28619
2016-07-15 10:58:25 -07:00
Matt Wringe
213c254bb7 Add in the ability to expose the ip address of a container. 2016-07-13 13:58:43 -04:00
Tim St. Clair
35f7bc5ee7 Move mocks to testing package to remove +build tags
Some go tools (e.g. godef, gorename) don't handle +build tags well, so
I refactored some packages to remove the test tags from cAdvisor.
2016-07-06 14:15:43 -07:00
Seth Jennings
eafff745a5 only fail fs stat gather if metadata device not found 2016-06-21 16:05:37 -05:00
Paul Morie
c37d048e1b Check for thin_ls binary in path when using ThinPoolWatcher 2016-06-14 13:19:05 -04:00
Tim St. Clair
3fbe18de9a Merge pull request #1204 from pmorie/dm-support
devicemapper thin_ls support
2016-05-18 12:30:05 -07:00
Paul Morie
647224c95a Add devicemapper support for docker containers 2016-05-18 15:19:20 -04:00
Shaya Potter
6fa3687720 Polling rkt implementation of new watcher inteface (#1284)
polling rkt implementation of new watcher inteface
2016-05-17 10:34:56 -07:00
Shaya Potter
e02632463b Refactor container watching out of raw handler into its own inteface / package 2016-05-11 20:27:10 -07:00
Tim St. Clair
8404284c42 Merge pull request #1274 from timstclair/container
Delete unused ListThreads container method
2016-05-09 11:59:28 -07:00
Tim St. Clair
d1868287db Clean up docker root dir 2016-05-06 19:31:36 -07:00
Tim St. Clair
4c506006f2 Don't validate docker state file, since it's no longer used 2016-05-06 19:29:24 -07:00
Tim St. Clair
019bfaecc7 Delete unused ListThreads container method 2016-05-06 13:45:29 -07:00
Tim St. Clair
05fb225624 Merge pull request #1269 from sjpotter/update-rkt
version checking for rkt + godep changes
2016-05-06 12:40:40 -07:00
Shaya Potter
44795d7ce0 add version checking for rkt api service + godep changes 2016-05-06 01:01:20 -07:00
Vish Kannan
bebe18b67a Merge pull request #1261 from sjpotter/listToCommon
move ListContainers code to common and make rkt play nice
2016-05-03 10:12:54 -07:00
Shaya Potter
5ca11bb1a0 move ListContainers code to common and make rkt play nice
playing nice means not elliding the system.slice cgroup but allowing the
raw handler to handle it
2016-05-03 07:14:53 -07:00
Tim St. Clair
4d3ef349fb Move utils/machine -> machine 2016-05-02 15:56:49 -07:00
Tim St. Clair
f365c6a115 Move docker types to v1 API 2016-05-02 15:52:29 -07:00
Tim St. Clair
0c89fd1b71 Refactor docker-specific functions from manager to docker 2016-05-02 12:24:31 -07:00
Lantao Liu
ece4c555cc switch to the new engine-api 2016-04-25 19:22:05 -07:00
derekwaynecarr
d01934a3e4 on systemd, we should ignore .mount cgroups 2016-04-20 23:47:19 -04:00
Shaya Potter
5dc9fbad3c Dont block on grpc if nothing is listeing on rkt api service port (#1218)
attempt to connect to rkt api service with net before grpc
2016-04-15 16:11:49 -07:00
Tim St. Clair
9790a0d3f4 Fix docker GetSpec to include image, labels, and env vars 2016-04-15 15:34:24 -07:00
Tim St. Clair
dc6415aef7 Check docker container existance the same way as raw & rkt 2016-04-15 11:35:31 -07:00
Tim St. Clair
4a8f3e4c93 Read docker container spec from cgroupfs, rather than libcontainer spec 2016-04-14 17:10:03 -07:00
Tim St. Clair
4861405904 Refactor common container GetSpec
- Pull out the root container cases, since they're only relevant in the
  raw container handler
- Pass parameters rather than depending on AbstractContainerInterface
2016-04-14 16:57:52 -07:00
Tim St. Clair
7b1820b1d4 Look for container state in containerd path 2016-04-13 15:09:08 -07:00
Tim St. Clair
b768a9d1dc If docker /info does not include ServerVersion, fallback to /version 2016-04-08 17:05:26 -07:00
Tim St. Clair
b553e02476 Fix cAdivsor docker validation 2016-04-08 17:05:26 -07:00
Tim St. Clair
d9c864324b Fix usage of the latest go-dockerclient 2016-04-04 18:01:47 -07:00
Shaya Potter
ee52fdf4d1 address jon and vish comments 2016-03-23 18:19:31 -07:00
Shaya Potter
206670a655 first cut of rkt handler 2016-03-21 17:34:42 -07:00
Vishnu kannan
e2717d8bb7 Avoid collecting network stats for non root cgroups in raw handler.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-03-15 12:16:11 -07:00
Shaya Potter
5e04a224ae pull out common parts of docker and raw container handlers for use by rkt handler 2016-03-02 16:12:04 -08:00
Vishnu Kannan
da888b11b2 Set spec.hasFilesystem to true by default for docker containers. 2016-03-01 09:58:28 -08:00
Seth Jennings
decac1ac15 nil pointer check 2016-02-29 11:37:25 -06:00
Vish Kannan
06ac85ca29 Merge pull request #1121 from vishh/opt-out-metrics
Support opt out for metrics.
2016-02-25 16:25:02 -08:00
Vishnu kannan
2defa0bddb Adding support for inodes.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-25 15:13:58 -08:00
Vishnu kannan
36415f465a Support opt out for metrics.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-24 15:57:31 -08:00
Vishnu kannan
18a2abc8e5 Ignore empty strings paths while calculting disk usage.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-24 11:04:10 -08:00
Vishnu kannan
07136d4e90 Do not fail stats collection if cpu_cfs_quota is not enabled.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-23 17:20:02 -08:00
Vishnu kannan
262ceb075c Fix docker storage path while calculating disk usage from within the container.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-18 16:44:33 -08:00
Vishnu kannan
e009e64663 Adding an exponential backoff for fs usage tracking using du
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-16 16:27:16 -08:00
Vishnu kannan
110540b4fe Timeout du after a minute.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-16 12:43:50 -08:00
Tim St. Clair
0bc286dc93 Stop leaking testing dependencies (and flags) in non-testing builds 2016-02-11 13:18:02 -08:00
Jimmi Dyson
31dcb5f3b7 Remove systemd docker container name check 2016-02-06 20:38:32 +00:00
Vishnu kannan
6a6abc19f4 remove unused var rwLayerIDDirTemplate
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-05 17:15:00 -08:00
Vishnu kannan
6f0c6d3667 add todo to cleanup variable names in docker handler
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-05 16:32:08 -08:00
Vishnu kannan
dabec0a6b2 Fix fs stats handling for non-aufs storage drivers.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-05 16:32:07 -08:00
Jimmi Dyson
a8946729de Merge pull request #969 from f0/master
add support for cgroup  CPUQuota  and CPUPeriod prometheus limit
2016-02-04 16:51:38 +00:00
Vishnu kannan
3f75344053 Fallback to flag defined value for storage dir in case docker info
doesn't provide the storage dir.

Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-03 18:00:59 -08:00
Vishnu kannan
f5829b4744 Rework the v2.1 API to expose container Info.
Added a test for filesystem stats. Devicemapper backend is ignored

Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-03 17:02:49 -08:00
Florian Koch
669bc4abfa add cgroup quota and period support / add missing tests 2016-02-02 20:33:11 +01:00
Vishnu kannan
59c03672d0 Handle fs metrics for docker version >= 1.10.0
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-02 10:20:20 -08:00
Vishnu kannan
5ab2f45fc6 Use storage dir from docker info for filesystem usage tracking.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-02-01 18:20:00 -08:00
Tim St. Clair
f5bceae3a2 Merge pull request #1033 from asteris-llc/master
Add the support for kafka in cAdvisor's storage, including output of container id and labels
2016-01-29 17:17:06 -05:00
Allison Richardet
80ba7e7db7 Add the support for kafka in cAdvisor's storage, including output of container id and labels
Addressing PR Feedback

Addressing PR Feedback
2016-01-27 21:33:57 -06:00
Jimmi Dyson
4c345b1989 Merge pull request #1063 from dqminh/fix-envs-handler
Initialize env mapping for labels when creating docker container
2016-01-27 10:46:24 +00:00
Vish Kannan
bef8522964 Merge pull request #1046 from jimmidyson/libcontainer-bump
bump(github.com/opencontainers/runc/libcontainer)
2016-01-26 14:35:46 -08:00
Jimmi Dyson
336821d28e Fix up networking stats for new docker network functionality 2016-01-26 11:32:39 +00:00
Jimmi Dyson
33386f899b bump(github.com/opencontainers/runc/libcontainer)
Fixes issues with breaking changes to ``GetPids` which is affecting
downstream consumers of cadvisor (e.g. Kubernetes).
2016-01-26 09:46:59 +00:00
Daniel Dao
127b09f5ae initialize env mapping
Signed-off-by: Daniel Dao <dqminh@cloudflare.com>
2016-01-18 15:58:45 +00:00
Vishnu Kannan
944201c86b Refactor api conversion methods. 2016-01-15 15:40:15 -08:00
Vishnu kannan
6dfdd8eca1 1. Update filesystem stats APIs.
2. Add BaseUsage to containers which includes only the rootfs usage, if available.

Signed-off-by: Vishnu kannan <vishnuk@google.com>
2016-01-15 15:39:17 -08:00
Shimin Guo
a26b58ec8e expose page cache size 2016-01-15 08:45:51 -08:00
Shimin Guo
1a867bdadd expose RSS 2016-01-15 08:45:51 -08:00
Jimmi Dyson
4e9d29a408 Fix FS usage goroutine leaks 2016-01-14 19:30:48 +00:00
Vish Kannan
01434a1c54 Merge pull request #1020 from miguel250/zfs-support
Add support ZFS filesystem
2016-01-13 11:52:37 -08:00
Daniel Dao
e5b6bfa94f add whitelisted env as container metadata
This add Envs to container spec as a metadata source. When using prometheus
exposition format, they will be merged into the list of metrics' labels.

Also changed the cli flag to docker_env_metadata_whitelist, and add refenrences
of whitelist envs to API

Signed-off-by: Daniel Dao <dqminh@cloudflare.com>
2016-01-13 11:22:13 +00:00
Florian Pfitzer
009761cb53 Export image name and env variables as prometheus labels 2016-01-05 10:49:20 +00:00
Miguel Perez
e0fef76668 Add support ZFS filesystem
- fix container usage stat not loading on first load
2015-12-15 15:41:19 -05:00
Vishnu kannan
a6daa760c8 Fix goroutine leak in docker fs handler logic.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2015-12-04 11:19:49 -08:00
Jonathan Boulle
4965f069b7 Fix case statements dealing with storageDriver
cf0adcc817 introduced two switch
statements to facilitate the addition of the `overlayStorageDriver`;
unfortunately neither of them conform to the Go switch semantic, which
does not fallthrough unless explicitly requested. In one case this was
innocuous (because a `break` was effectively the same as a no-op) but in
the other it would cause the `HasFilesystem` bool to not be set
appropriately in the case of `aufsStorageDriver` being used.

IMHO it's also more idiomatic to perform the default behaviour in the
default case rather than pre-setting and overriding it.
2015-12-02 16:04:01 -08:00
Vishnu Kannan
cf0adcc817 Add support for Overlayfs.
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2015-12-02 15:43:34 -08:00
Lei Xue
15b34b0131 add test case for compatibility.go 2015-12-02 11:01:50 +08:00
Lei Xue
7343ae4583 fix unmarshal container config failure with Docker 1.8.3 2015-12-02 11:01:12 +08:00
Lei Xue
dbbe38dfed re-order the import package 2015-11-30 16:43:22 +08:00
Jimmi Dyson
1f679cee70 Clean up unused struct fields (via structcheck linter) 2015-11-27 22:06:16 +00:00
Jimmi Dyson
82810f13cd Remove unused code (via deadcode linter) 2015-11-27 21:48:33 +00:00
Jimmi Dyson
360c73c6fd Improve perf of interface stats parsing 2015-11-27 14:12:41 +00:00
Jimmi Dyson
f9eb56e800 Merge pull request #966 from afein/godep_update_runc
[Godeps] changed docker/libcontainer dependency to runc/libcontainer
2015-11-26 15:19:28 +00:00
Jimmi Dyson
cec96eb68b Merge pull request #978 from jimmidyson/regexp-perf
Regexp tidy up
2015-11-26 09:45:27 +00:00
Jimmi Dyson
d1fce20304 Regexp tidy up 2015-11-26 09:14:26 +00:00
Jimmi Dyson
17622ecba1 Rename readInt64 -> readUInt64 for clarity 2015-11-25 16:42:33 +00:00
Alex Mavrogiannis
4533dd7d18 changed libcontainer dependency to runc 2015-11-21 14:04:01 -08:00
Vishnu kannan
401cbbc48f Add a --nosystemd flag to avoid assuming systemd to be the cgroups
owner for docker containers.

Signed-off-by: Vishnu kannan <vishnuk@google.com>
2015-11-16 10:37:54 -08:00
Vishnu Kannan
419dd8a778 Include log usage for aufs driver.
Signed-off-by: Vishnu Kannan <vishnuk@google.com>
2015-11-10 14:16:20 -08:00
Vish Kannan
47a1fa4fff Merge pull request #936 from mqliang/master
Fix imported package names to not use mixedCaps or under_scores
2015-11-04 10:47:48 -08:00
Jimmi Dyson
561cc1da4f Use file reader directly for net stats 2015-10-28 12:51:19 +00:00
Jimmi Dyson
c72e0c23a5 Add test for net dev stats 2015-10-28 12:51:13 +00:00
Jimmi Dyson
da771a0977 Drop regexp for net stats parsing
Reported in kubernetes/kubernetes#16296
2015-10-27 20:16:49 +00:00
Jimmi Dyson
8b6e002e0a Disable tcp stats collection
Fixes #938
2015-10-22 21:05:46 +01:00
mqliang
ce001dcd4e Fix imported package names to not use mixedCaps or under_scores 2015-10-22 12:10:57 +08:00
Jimmi Dyson
b8b6e1bac1 Support devicemapper storage for docker images dir
Fixes #920
2015-10-21 09:56:33 +01:00
Jimmi Dyson
357e6a0f23 Reuse cached docker client
Fixes #925
2015-10-20 20:15:54 +01:00
Jimmi Dyson
5a5d0575f5 Docker, libcontainer, docker client bumps 2015-10-20 09:22:12 +01:00
Jimmi Dyson
b4a6819291 Switch to select on time.After & close channel for track usage 2015-10-14 09:58:42 +01:00
Vishnu kannan
dd8345ab32 Perform separate housekeeping for docker container filesystem stats. 2015-10-08 15:58:57 -07:00
Vishnu kannan
6e51487236 Compute fs usage for docker containers less often.
`du` can cause a lot of cpu usage.
2015-10-08 15:01:57 -07:00
Jimmi Dyson
6ee904d129 Merge pull request #821 from basvdlei/ignore
Reduce the amount of bad Docker.inspects when using docker_only flag in combination with Systemd
2015-10-02 15:31:43 +01:00
Tomas Kral
bd61caf0c3 add failcnt 2015-10-02 14:24:22 +02:00
Bas van der Lei
f8eb8cc982 validate name with isContainerName func 2015-10-02 13:49:27 +02:00
Bas van der Lei
9931854585 remove dead code 2015-10-02 13:42:40 +02:00
Bas van der Lei
13674cf06c optimize dockerCgroupRegexp 2015-09-27 00:20:52 +02:00
Florian Koch
e4262b91b1 move TCP and TCP6 stats to NetworkStats 2015-09-25 09:04:53 +02:00
Florian Koch
dd041457b5 some fixes 2015-09-24 15:44:42 +02:00
Florian Koch
c331982f21 add tcp/tcp6 statistics 2015-09-24 15:44:42 +02:00
Bas van der Lei
7788c8bd27 Ignore systemd 'containers' in the docker driver 2015-09-23 23:55:35 +02:00
Jimmi Dyson
b8fc8cd2ae Add container specs & cadvisor version info to prometheus metrics 2015-09-09 14:28:54 +01:00
Jimmi Dyson
7e10398a50 Use proc fs to get network stats.
Reasons discussed in
https://github.com/google/cadvisor/issues/822#issuecomment-135811901 &
following.
2015-08-29 00:20:07 +01:00
Jimmi Dyson
ff0796712b Add image name to container spec & prometheus metrics
Fixes #848
2015-08-28 12:38:16 +01:00
Jimmi Dyson
d5fa97c998 Get network stats by switching network namespace on newer Docker
versions.

Fixes #822
2015-08-25 23:27:01 +01:00
Rohit Jnagal
7a2f508b50 Fix network info for docker containers running with --net=host.
Docker still reports a loop device for these. Need to check for
more than one device to mark network as available.
2015-08-24 02:58:03 +00:00
Rohit Jnagal
a123fd72d8 Add logic to read custom metric config files from container root.
Docker does not provide the rootfs path through docker inspect or statefile
and the path is dependent on the storage driver being used.

Instead of enumerating the storage drivers, we pick a pid from the container
and get the config from /proc/pid/root. Although a bit expensive, this method
works for non-docker containers too.
2015-07-22 15:45:07 +00:00
Victor Marmol
4a9a57cdd0 Merge pull request #814 from anushree-n/typo
Fix typo
2015-07-13 10:41:01 -07:00
anushree-n
fff8880a8d Typo in helpers.go 2015-07-13 10:30:46 -07:00