prometheus/cadvisor
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Migrating cAdvisor code from lmctfy

Browse Source
This commit is contained in:
Victor Marmol 2014-06-09 12:12:07 -07:00
parent cc8fc1fcdd
commit 712f9fb32f
46 changed files with 6925 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored Normal file
View File

@ -0,0 +1 @@
*.swp

11
AUTHORS Normal file
View File

@ -0,0 +1,11 @@
# This is the official list of cAdvisor authors for copyright purposes.
# This file is distinct from the CONTRIBUTORS files.
# See the latter for an explanation.
# Names should be added to this file as
# Name or Organization <email address>
# The email address is not required for organizations.
# Please keep the list sorted.
Google Inc.

11
CONTRIBUTORS Normal file
View File

@ -0,0 +1,11 @@
# This is the official list of people who have contributed to the project. The
# copyright is held by those individuals or organizations in the AUTHORS file.
#
# Names should be added to this file like so:
# Name <email address>
# Please keep the list sorted by first name.
Kamil Yurtsever <kyurtsever@google.com>
Nan Deng <dengnan@google.com>
Victor Marmol <vmarmol@google.com>

190
LICENSE Normal file
View File

@ -0,0 +1,190 @@
Copyright 2014 The cAdvisor Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS

2
TODO.txt Normal file
View File

@ -0,0 +1,2 @@
- Allow us to have different tracking policies: only top-level, only specified, all containers
- Add ability to checkpoint state (and plugable medium for this)

88
api/handler.go Normal file
View File

@ -0,0 +1,88 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Handler for /api/
package api
import (
"encoding/json"
"fmt"
"log"
"net/http"
"net/url"
"strings"
"time"
"github.com/google/cadvisor/manager"
)
const (
ApiResource = "/api/v1.0/"
ContainersApi = "containers"
MachineApi = "machine"
)
func HandleRequest(m manager.Manager, w http.ResponseWriter, u *url.URL) error {
start := time.Now()
// Get API request type.
requestType := u.Path[len(ApiResource):]
i := strings.Index(requestType, "/")
requestArgs := ""
if i != -1 {
requestArgs = requestType[i:]
requestType = requestType[:i]
}
if requestType == MachineApi {
log.Printf("Api - Machine")
// Get the MachineInfo
machineInfo, err := m.GetMachineInfo()
if err != nil {
return err
}
out, err := json.Marshal(machineInfo)
if err != nil {
fmt.Fprintf(w, "Failed to marshall MachineInfo with error: %s", err)
}
w.Write(out)
} else if requestType == ContainersApi {
// The container name is the path after the requestType
containerName := requestArgs
log.Printf("Api - Container(%s)", containerName)
// Get the container.
cont, err := m.GetContainerInfo(containerName)
if err != nil {
fmt.Fprintf(w, "Failed to get container \"%s\" with error: %s", containerName, err)
return err
}
// Only output the container as JSON.
out, err := json.Marshal(cont)
if err != nil {
fmt.Fprintf(w, "Failed to marshall container %q with error: %s", containerName, err)
}
w.Write(out)
} else {
return fmt.Errorf("unknown API request type %q", requestType)
}
log.Printf("Request took %s", time.Since(start))
return nil
}

98
cadvisor.go Normal file
View File

@ -0,0 +1,98 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"flag"
"fmt"
"log"
"net/http"
"time"
"github.com/google/cadvisor/api"
"github.com/google/cadvisor/container"
"github.com/google/cadvisor/container/docker"
"github.com/google/cadvisor/container/lmctfy"
"github.com/google/cadvisor/manager"
"github.com/google/cadvisor/pages"
"github.com/google/cadvisor/pages/static"
)
var argPort = flag.Int("port", 8080, "port to listen")
var argSampleSize = flag.Int("samples", 1024, "number of samples we want to keep")
var argResetPeriod = flag.Duration("reset_period", 2*time.Hour, "period to reset the samples")
func main() {
flag.Parse()
// XXX(dengnan): Should we allow users to specify which sampler they want to use?
container.SetStatsParameter(&container.StatsParameter{
Sampler: "uniform",
NumSamples: *argSampleSize,
ResetPeriod: *argResetPeriod,
})
containerManager, err := manager.New()
if err != nil {
log.Fatalf("Failed to create a Container Manager: %s", err)
}
if err := lmctfy.Register("/"); err != nil {
log.Printf("lmctfy registration failed: %v.", err)
log.Print("Running in docker only mode.")
if err := docker.Register(containerManager, "/"); err != nil {
log.Printf("Docker registration failed: %v.", err)
log.Fatalf("Unable to continue without docker or lmctfy.")
}
}
if err := docker.Register(containerManager, "/docker"); err != nil {
// Ignore this error because we should work with lmctfy only
log.Printf("Docker registration failed: %v.", err)
log.Print("Running in lmctfy only mode.")
}
// Handler for static content.
http.HandleFunc(static.StaticResource, func(w http.ResponseWriter, r *http.Request) {
err := static.HandleRequest(w, r.URL)
if err != nil {
fmt.Fprintf(w, "%s", err)
}
})
// Handler for the API.
http.HandleFunc(api.ApiResource, func(w http.ResponseWriter, r *http.Request) {
err := api.HandleRequest(containerManager, w, r.URL)
if err != nil {
fmt.Fprintf(w, "%s", err)
}
})
// Redirect / to containers page.
http.Handle("/", http.RedirectHandler(pages.ContainersPage, http.StatusTemporaryRedirect))
// Register the handler for the containers page.
http.HandleFunc(pages.ContainersPage, func(w http.ResponseWriter, r *http.Request) {
err := pages.ServerContainersPage(containerManager, w, r.URL)
if err != nil {
fmt.Fprintf(w, "%s", err)
}
})
go containerManager.Start()
log.Print("About to serve on port ", *argPort)
addr := fmt.Sprintf(":%v", *argPort)
log.Fatal(http.ListenAndServe(addr, nil))
}

96
client/client.go Normal file
View File

@ -0,0 +1,96 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cadvisor
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"strings"
"github.com/google/cadvisor/info"
)
type Client struct {
baseUrl string
}
func NewClient(URL string) (*Client, error) {
c := &Client{
baseUrl: strings.Join([]string{
URL,
"api/v1.0",
}, "/"),
}
_, err := c.MachineInfo()
if err != nil {
return nil, err
}
return c, nil
}
func (self *Client) machineInfoUrl() string {
return strings.Join([]string{self.baseUrl, "machine"}, "/")
}
func (self *Client) MachineInfo() (minfo *info.MachineInfo, err error) {
u := self.machineInfoUrl()
ret := new(info.MachineInfo)
err = self.httpGetJsonData(ret, u, "machine info")
if err != nil {
return
}
minfo = ret
return
}
func (self *Client) containerInfoUrl(name string) string {
if name[0] == '/' {
name = name[1:]
}
return strings.Join([]string{self.baseUrl, "containers", name}, "/")
}
func (self *Client) httpGetJsonData(data interface{}, url, infoName string) error {
resp, err := http.Get(url)
if err != nil {
err = fmt.Errorf("unable to get %v: %v", infoName, err)
return err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
err = fmt.Errorf("unable to read all %v: %v", infoName, err)
return err
}
err = json.Unmarshal(body, data)
if err != nil {
err = fmt.Errorf("unable to unmarshal %v (%v): %v", infoName, string(body), err)
return err
}
return nil
}
func (self *Client) ContainerInfo(name string) (cinfo *info.ContainerInfo, err error) {
u := self.containerInfoUrl(name)
ret := new(info.ContainerInfo)
err = self.httpGetJsonData(ret, u, fmt.Sprintf("container info for %v", name))
if err != nil {
return
}
cinfo = ret
return
}

220
client/client_test.go Normal file
View File

@ -0,0 +1,220 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cadvisor
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"reflect"
"testing"
"github.com/google/cadvisor/info"
)
func testGetJsonData(
strRep string,
emptyData interface{},
f func() (interface{}, error),
) error {
err := json.Unmarshal([]byte(strRep), emptyData)
if err != nil {
return fmt.Errorf("invalid json input: %v", err)
}
reply, err := f()
if err != nil {
return fmt.Errorf("unable to retrieve data: %v", err)
}
if !reflect.DeepEqual(reply, emptyData) {
return fmt.Errorf("retrieved wrong data: %+v != %+v", reply, emptyData)
}
return nil
}
func cadvisorTestClient(path, reply string) (*Client, *httptest.Server, error) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == path {
fmt.Fprint(w, reply)
} else if r.URL.Path == "/api/v1.0/machine" {
fmt.Fprint(w, `{"num_cores":8,"memory_capacity":31625871360}`)
} else {
w.WriteHeader(http.StatusNotFound)
fmt.Fprintf(w, "Page not found.")
}
}))
client, err := NewClient(ts.URL)
if err != nil {
ts.Close()
return nil, nil, err
}
return client, ts, err
}
func TestGetMachineinfo(t *testing.T) {
respStr := `{"num_cores":8,"memory_capacity":31625871360}`
client, server, err := cadvisorTestClient("/api/v1.0/machine", respStr)
if err != nil {
t.Fatalf("unable to get a client %v", err)
}
defer server.Close()
err = testGetJsonData(respStr, &info.MachineInfo{}, func() (interface{}, error) {
return client.MachineInfo()
})
if err != nil {
t.Fatal(err)
}
}
func TestGetContainerInfo(t *testing.T) {
respStr := `
{
"name": "%v",
"subcontainers": [
"%v/cadvisor",
"%v/docker"
],
"spec": {
"cpu": {
"limit": 1000,
"max_limit": 18446744073709551000,
"mask": {
"data": [
255
]
}
},
"memory": {
"limit": 9223372036854776000,
"reservation": 9223372036854776000
}
},
"stats": [
{
"timestamp": "2014-06-02T19:04:37.112952602Z",
"cpu": {
"usage": {
"total": 39238585820907,
"per_cpu": [
6601830440753,
5399612248434,
4705072360704,
4516301677099,
4585779804509,
4556599077540,
4479432306284,
4393957905584
],
"user": 14513300000000,
"system": 14484560000000
},
"load": 0
},
"memory": {
"limit": 9223372036854776000,
"usage": 837984256,
"container_data": {
"pgfault": 71601418,
"pgmajfault": 664
},
"hierarchical_data": {
"pgfault": 100135740,
"pgmajfault": 1454
}
}
},
{
"timestamp": "2014-06-02T19:04:38.117962404Z",
"cpu": {
"usage": {
"total": 39238651066794,
"per_cpu": [
6601838963405,
5399619233761,
4705084549250,
4516308300389,
4585786473026,
4556607159034,
4479432477487,
4393973910442
],
"user": 14513320000000,
"system": 14484590000000
},
"load": 0
},
"memory": {
"limit": 9223372036854776000,
"usage": 838025216,
"container_data": {
"pgfault": 71601418,
"pgmajfault": 664
},
"hierarchical_data": {
"pgfault": 100140566,
"pgmajfault": 1454
}
}
},
{
"timestamp": "2014-06-02T19:04:39.122826983Z",
"cpu": {
"usage": {
"total": 39238719219625,
"per_cpu": [
6601852483847,
5399630546695,
4705091917507,
4516318924052,
4585791208645,
4556612305230,
4479432924370,
4393988909279
],
"user": 14513340000000,
"system": 14484630000000
},
"load": 0
},
"memory": {
"limit": 9223372036854776000,
"usage": 838131712,
"container_data": {
"pgfault": 71601418,
"pgmajfault": 664
},
"hierarchical_data": {
"pgfault": 100145700,
"pgmajfault": 1454
}
}
}
]
}
`
containerName := "/some/container"
respStr = fmt.Sprintf(respStr, containerName, containerName, containerName)
client, server, err := cadvisorTestClient(fmt.Sprintf("/api/v1.0/containers%v", containerName), respStr)
if err != nil {
t.Fatalf("unable to get a client %v", err)
}
defer server.Close()
err = testGetJsonData(respStr, &info.ContainerInfo{}, func() (interface{}, error) {
return client.ContainerInfo(containerName)
})
if err != nil {
t.Fatal(err)
}
}

46
container/container.go Normal file
View File

@ -0,0 +1,46 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package container
import (
"fmt"
"github.com/google/cadvisor/info"
)
// Listing types.
const (
LIST_SELF = iota
LIST_RECURSIVE
)
type ListType int
// Interface for container operation handlers.
type ContainerHandler interface {
GetSpec() (*info.ContainerSpec, error)
GetStats() (*info.ContainerStats, error)
ListContainers(listType ListType) ([]string, error)
ListThreads(listType ListType) ([]int, error)
ListProcesses(listType ListType) ([]int, error)
StatsSummary() (*info.ContainerStatsSummary, error)
}
type NoStatsSummary struct {
}
func (self *NoStatsSummary) StatsSummary() (*info.ContainerStatsSummary, error) {
return nil, fmt.Errorf("This method (StatsSummary) should never be called")
}

104
container/docker/factory.go Normal file
View File

@ -0,0 +1,104 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package docker
import (
"flag"
"fmt"
"regexp"
"strconv"
"github.com/fsouza/go-dockerclient"
"github.com/google/cadvisor/container"
"github.com/google/cadvisor/info"
)
var argDockerEndpoint = flag.String("docker", "unix:///var/run/docker.sock", "docker endpoint")
type dockerFactory struct {
machineInfoFactory info.MachineInfoFactory
}
func (self *dockerFactory) String() string {
return "docker"
}
func (self *dockerFactory) NewContainerHandler(name string) (handler container.ContainerHandler, err error) {
client, err := docker.NewClient(*argDockerEndpoint)
if err != nil {
return
}
handler = &dockerContainerHandler{
client: client,
name: name,
machineInfoFactory: self.machineInfoFactory,
}
return
}
func parseDockerVersion(full_version_string string) ([]int, error) {
version_regexp_string := "(\\d+)\\.(\\d+)\\.(\\d+)"
version_re := regexp.MustCompile(version_regexp_string)
matches := version_re.FindAllStringSubmatch(full_version_string, -1)
if len(matches) != 1 {
return nil, fmt.Errorf("Version string \"%v\" doesn't match expected regular expression: \"%v\"", full_version_string, version_regexp_string)
}
version_string_array := matches[0][1:]
version_array := make([]int, 3)
for index, version_string := range version_string_array {
version, err := strconv.Atoi(version_string)
if err != nil {
return nil, fmt.Errorf("Error while parsing \"%v\" in \"%v\"", version_string, full_version_string)
}
version_array[index] = version
}
return version_array, nil
}
// Register root container before running this function!
func Register(factory info.MachineInfoFactory, paths ...string) error {
client, err := docker.NewClient(*argDockerEndpoint)
if err != nil {
return fmt.Errorf("unable to communicate with docker daemon: %v", err)
}
if version, err := client.Version(); err != nil {
return fmt.Errorf("unable to communicate with docker daemon: %v", err)
} else {
expected_version := []int{0, 11, 1}
version_string := version.Get("Version")
version, err := parseDockerVersion(version_string)
if err != nil {
return fmt.Errorf("Couldn't parse docker version: %v", err)
}
for index, number := range version {
if number > expected_version[index] {
break
} else if number < expected_version[index] {
return fmt.Errorf("cAdvisor requires docker version above %v but we have found version %v reported as \"%v\"", expected_version, version, version_string)
}
}
}
f := &dockerFactory{
machineInfoFactory: factory,
}
cf := container.AddStatsSummaryToFactory(f)
for _, p := range paths {
if p != "/" && p != "/docker" {
return fmt.Errorf("%v cannot be managed by docker", p)
}
container.RegisterContainerHandlerFactory(p, cf)
}
return nil
}

319
container/docker/handler.go Normal file
View File

@ -0,0 +1,319 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package docker
import (
"bufio"
"encoding/json"
"fmt"
"math"
"os"
"path"
"strings"
"time"
"github.com/dotcloud/docker/nat"
"github.com/dotcloud/docker/pkg/libcontainer/cgroups"
"github.com/dotcloud/docker/pkg/libcontainer/cgroups/fs"
"github.com/fsouza/go-dockerclient"
"github.com/google/cadvisor/container"
"github.com/google/cadvisor/info"
)
type dockerContainerHandler struct {
client *docker.Client
name string
machineInfoFactory info.MachineInfoFactory
container.NoStatsSummary
}
func (self *dockerContainerHandler) splitName() (string, string, error) {
parent, id := path.Split(self.name)
cgroupSelf, err := os.Open("/proc/self/cgroup")
if err != nil {
return "", "", err
}
scanner := bufio.NewScanner(cgroupSelf)
subsys := []string{"memory", "cpu"}
nestedLevels := 0
for scanner.Scan() {
line := scanner.Text()
elems := strings.Split(line, ":")
if len(elems) < 3 {
continue
}
for _, s := range subsys {
if elems[1] == s {
// count how many nested docker containers are there.
nestedLevels = strings.Count(elems[2], "/docker")
break
}
}
}
if nestedLevels > 0 {
// we are running inside a docker container
upperLevel := strings.Repeat("../../", nestedLevels)
//parent = strings.Join([]string{parent, upperLevel}, "/")
parent = fmt.Sprintf("%v%v", upperLevel, parent)
}
return parent, id, nil
}
func (self *dockerContainerHandler) isDockerRoot() bool {
// TODO(dengnan): Should we consider other cases?
return self.name == "/docker"
}
func (self *dockerContainerHandler) isRootContainer() bool {
return self.name == "/"
}
func (self *dockerContainerHandler) isDockerContainer() bool {
return (!self.isDockerRoot()) && (!self.isRootContainer())
}
type dockerPortBinding struct {
HostIp string
HostPort string
}
type dockerPort string
type dockerPortMap map[dockerPort][]dockerPortBinding
type dockerNetworkSettings struct {
IPAddress string
IPPrefixLen int
Gateway string
Bridge string
Ports dockerPortMap
}
type dockerContainerConfig struct {
Hostname string
Domainname string
User string
Memory int64 // Memory limit (in bytes)
MemorySwap int64 // Total memory usage (memory + swap); set `-1' to disable swap
CpuShares int64 // CPU shares (relative weight vs. other containers)
Cpuset string // Cpuset 0-2, 0,1
AttachStdin bool
AttachStdout bool
AttachStderr bool
PortSpecs []string // Deprecated - Can be in the format of 8080/tcp
ExposedPorts map[nat.Port]struct{}
Tty bool // Attach standard streams to a tty, including stdin if it is not closed.
OpenStdin bool // Open stdin
StdinOnce bool // If true, close stdin after the 1 attached client disconnects.
Env []string
Cmd []string
Image string // Name of the image as it was passed by the operator (eg. could be symbolic)
Volumes map[string]struct{}
WorkingDir string
Entrypoint []string
NetworkDisabled bool
OnBuild []string
}
type dockerState struct {
Running bool
Pid int
ExitCode int
StartedAt time.Time
FinishedAt time.Time
}
type dockerContainerSpec struct {
ID string
Created time.Time
Path string
Args []string
Config *dockerContainerConfig
State dockerState
Image string
NetworkSettings *dockerNetworkSettings
ResolvConfPath string
HostnamePath string
HostsPath string
Name string
Driver string
ExecDriver string
MountLabel, ProcessLabel string
Volumes map[string]string
// Store rw/ro in a separate structure to preserve reverse-compatibility on-disk.
// Easier than migrating older container configs :)
VolumesRW map[string]bool
// contains filtered or unexported fields
}
func readDockerSpec(id string) (spec *dockerContainerSpec, err error) {
dir := "/var/lib/docker/containers"
configPath := path.Join(dir, id, "config.json")
f, err := os.Open(configPath)
if err != nil {
return
}
defer f.Close()
d := json.NewDecoder(f)
ret := new(dockerContainerSpec)
err = d.Decode(ret)
if err != nil {
return
}
spec = ret
return
}
func dockerConfigToContainerSpec(config *dockerContainerSpec, mi *info.MachineInfo) *info.ContainerSpec {
spec := new(info.ContainerSpec)
spec.Memory = new(info.MemorySpec)
spec.Memory.Limit = math.MaxUint64
spec.Memory.SwapLimit = math.MaxUint64
if config.Config.Memory > 0 {
spec.Memory.Limit = uint64(config.Config.Memory)
}
if config.Config.MemorySwap > 0 {
spec.Memory.SwapLimit = uint64(config.Config.MemorySwap - config.Config.Memory)
}
if mi != nil {
spec.Cpu = new(info.CpuSpec)
spec.Cpu.Limit = math.MaxUint64
n := mi.NumCores / 64
if mi.NumCores%64 > 0 {
n++
}
spec.Cpu.Mask.Data = make([]uint64, n)
for i := 0; i < n; i++ {
spec.Cpu.Mask.Data[i] = math.MaxUint64
}
}
return spec
}
func (self *dockerContainerHandler) GetSpec() (spec *info.ContainerSpec, err error) {
if !self.isDockerContainer() {
spec = new(info.ContainerSpec)
return
}
mi, err := self.machineInfoFactory.GetMachineInfo()
if err != nil {
return
}
_, id, err := self.splitName()
if err != nil {
return
}
dspec, err := readDockerSpec(id)
if err != nil {
return
}
spec = dockerConfigToContainerSpec(dspec, mi)
return
}
func libcontainerToContainerStats(s *cgroups.Stats, mi *info.MachineInfo) *info.ContainerStats {
ret := new(info.ContainerStats)
ret.Timestamp = time.Now()
ret.Cpu = new(info.CpuStats)
ret.Cpu.Usage.User = s.CpuStats.CpuUsage.UsageInUsermode
ret.Cpu.Usage.System = s.CpuStats.CpuUsage.UsageInKernelmode
n := len(s.CpuStats.CpuUsage.PercpuUsage)
ret.Cpu.Usage.PerCpu = make([]uint64, n)
ret.Cpu.Usage.Total = 0
for i := 0; i < n; i++ {
ret.Cpu.Usage.PerCpu[i] = s.CpuStats.CpuUsage.PercpuUsage[i]
ret.Cpu.Usage.Total += s.CpuStats.CpuUsage.PercpuUsage[i]
}
ret.Memory = new(info.MemoryStats)
ret.Memory.Usage = s.MemoryStats.Usage
if v, ok := s.MemoryStats.Stats["pgfault"]; ok {
ret.Memory.ContainerData.Pgfault = v
ret.Memory.HierarchicalData.Pgfault = v
}
if v, ok := s.MemoryStats.Stats["pgmajfault"]; ok {
ret.Memory.ContainerData.Pgmajfault = v
ret.Memory.HierarchicalData.Pgmajfault = v
}
return ret
}
func (self *dockerContainerHandler) GetStats() (stats *info.ContainerStats, err error) {
if !self.isDockerContainer() {
return
}
mi, err := self.machineInfoFactory.GetMachineInfo()
if err != nil {
return
}
parent, id, err := self.splitName()
if err != nil {
return
}
cg := &cgroups.Cgroup{
Parent: parent,
Name: id,
}
s, err := fs.GetStats(cg)
if err != nil {
return
}
stats = libcontainerToContainerStats(s, mi)
return
}
func (self *dockerContainerHandler) ListContainers(listType container.ListType) ([]string, error) {
if self.isDockerContainer() {
return nil, nil
}
if self.isRootContainer() && listType == container.LIST_SELF {
return []string{"/docker"}, nil
}
opt := docker.ListContainersOptions{
All: true,
}
containers, err := self.client.ListContainers(opt)
if err != nil {
return nil, err
}
ret := make([]string, 0, len(containers)+1)
for _, c := range containers {
if !strings.HasPrefix(c.Status, "Up ") {
continue
}
path := fmt.Sprintf("/docker/%v", c.ID)
ret = append(ret, path)
}
if self.isRootContainer() {
ret = append(ret, "/docker")
}
return ret, nil
}
func (self *dockerContainerHandler) ListThreads(listType container.ListType) ([]int, error) {
return nil, nil
}
func (self *dockerContainerHandler) ListProcesses(listType container.ListType) ([]int, error) {
return nil, nil
}

129
container/factory.go Normal file
View File

@ -0,0 +1,129 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package container
import (
"fmt"
"log"
"strings"
"sync"
)
type ContainerHandlerFactory interface {
NewContainerHandler(name string) (ContainerHandler, error)
// for testability
String() string
}
type factoryTreeNode struct {
defaultFactory ContainerHandlerFactory
children map[string]*factoryTreeNode
}
func (self *factoryTreeNode) find(elems ...string) ContainerHandlerFactory {
node := self
for _, elem := range elems {
if len(node.children) == 0 {
break
}
if child, ok := node.children[elem]; ok {
node = child
} else {
return node.defaultFactory
}
}
return node.defaultFactory
}
func (self *factoryTreeNode) add(factory ContainerHandlerFactory, elems ...string) {
node := self
for _, elem := range elems {
if node.children == nil {
node.children = make(map[string]*factoryTreeNode, 16)
}
child, ok := self.children[elem]
if !ok {
child = &factoryTreeNode{
defaultFactory: node.defaultFactory,
children: make(map[string]*factoryTreeNode, 16),
}
node.children[elem] = child
}
node = child
}
node.defaultFactory = factory
}
type factoryManager struct {
root *factoryTreeNode
lock sync.RWMutex
}
func dropEmptyString(elems ...string) []string {
ret := make([]string, 0, len(elems))
for _, e := range elems {
if len(e) > 0 {
ret = append(ret, e)
}
}
return ret
}
// Must register factory for root container!
func (self *factoryManager) Register(path string, factory ContainerHandlerFactory) {
self.lock.Lock()
defer self.lock.Unlock()
if self.root == nil {
self.root = &factoryTreeNode{
defaultFactory: nil,
children: make(map[string]*factoryTreeNode, 10),
}
}
elems := dropEmptyString(strings.Split(path, "/")...)
self.root.add(factory, elems...)
}
func (self *factoryManager) NewContainerHandler(path string) (ContainerHandler, error) {
self.lock.RLock()
defer self.lock.RUnlock()
if self.root == nil {
err := fmt.Errorf("nil factory for container %v: no factory registered", path)
return nil, err
}
elems := dropEmptyString(strings.Split(path, "/")...)
factory := self.root.find(elems...)
if factory == nil {
err := fmt.Errorf("nil factory for container %v", path)
return nil, err
}
log.Printf("container handler factory for %v is %v\n", path, factory)
return factory.NewContainerHandler(path)
}
var globalFactoryManager factoryManager
func RegisterContainerHandlerFactory(path string, factory ContainerHandlerFactory) {
globalFactoryManager.Register(path, factory)
}
func NewContainerHandler(path string) (ContainerHandler, error) {
return globalFactoryManager.NewContainerHandler(path)
}

76
container/factory_test.go Normal file
View File

@ -0,0 +1,76 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package container
import (
"strings"
"testing"
"github.com/stretchr/testify/mock"
)
type mockContainerHandlerFactory struct {
mock.Mock
Name string
}
func (self *mockContainerHandlerFactory) String() string {
return self.Name
}
func (self *mockContainerHandlerFactory) NewContainerHandler(name string) (ContainerHandler, error) {
args := self.Called(name)
return args.Get(0).(ContainerHandler), args.Error(1)
}
func testExpectedFactory(root *factoryTreeNode, path, expectedFactory string, t *testing.T) {
elems := dropEmptyString(strings.Split(path, "/")...)
factory := root.find(elems...)
if factory.String() != expectedFactory {
t.Errorf("factory %v should be used to create container %v. but %v is selected",
expectedFactory,
path,
factory)
}
}
func testAddFactory(root *factoryTreeNode, path string) *factoryTreeNode {
elems := dropEmptyString(strings.Split(path, "/")...)
if root == nil {
root = &factoryTreeNode{
defaultFactory: nil,
}
}
f := &mockContainerHandlerFactory{
Name: path,
}
root.add(f, elems...)
return root
}
func TestFactoryTree(t *testing.T) {
root := testAddFactory(nil, "/")
root = testAddFactory(root, "/docker")
root = testAddFactory(root, "/user")
root = testAddFactory(root, "/user/special/containers")
testExpectedFactory(root, "/docker/container", "/docker", t)
testExpectedFactory(root, "/docker", "/docker", t)
testExpectedFactory(root, "/", "/", t)
testExpectedFactory(root, "/user/deep/level/container", "/user", t)
testExpectedFactory(root, "/user/special/containers", "/user/special/containers", t)
testExpectedFactory(root, "/user/special/containers/container", "/user/special/containers", t)
testExpectedFactory(root, "/other", "/", t)
}

90
container/filter.go Normal file
View File

@ -0,0 +1,90 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package container
import (
"strings"
"github.com/google/cadvisor/info"
)
type containerListFilter struct {
filter func(string) bool
handler ContainerHandler
NoStatsSummary
}
func (self *containerListFilter) GetSpec() (*info.ContainerSpec, error) {
return self.handler.GetSpec()
}
func (self *containerListFilter) GetStats() (*info.ContainerStats, error) {
return self.handler.GetStats()
}
func (self *containerListFilter) ListContainers(listType ListType) ([]string, error) {
containers, err := self.handler.ListContainers(listType)
if err != nil {
return nil, err
}
if len(containers) == 0 {
return nil, nil
}
ret := make([]string, 0, len(containers))
for _, c := range containers {
if self.filter(c) {
ret = append(ret, c)
}
}
return ret, nil
}
func (self *containerListFilter) ListThreads(listType ListType) ([]int, error) {
return self.handler.ListThreads(listType)
}
func (self *containerListFilter) ListProcesses(listType ListType) ([]int, error) {
return self.handler.ListProcesses(listType)
}
func NewWhiteListFilter(handler ContainerHandler, acceptedPaths ...string) ContainerHandler {
filter := func(p string) bool {
for _, path := range acceptedPaths {
if strings.HasPrefix(p, path) {
return true
}
}
return false
}
return &containerListFilter{
filter: filter,
handler: handler,
}
}
func NewBlackListFilter(handler ContainerHandler, forbiddenPaths ...string) ContainerHandler {
filter := func(p string) bool {
for _, path := range forbiddenPaths {
if strings.HasPrefix(p, path) {
return false
}
}
return true
}
return &containerListFilter{
filter: filter,
handler: handler,
}
}

123
container/filter_test.go Normal file
View File

@ -0,0 +1,123 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package container
import (
"strings"
"testing"
"github.com/google/cadvisor/info"
"github.com/stretchr/testify/mock"
)
type mockContainerHandler struct {
mock.Mock
NoStatsSummary
}
func (self *mockContainerHandler) GetSpec() (*info.ContainerSpec, error) {
args := self.Called()
return args.Get(0).(*info.ContainerSpec), args.Error(1)
}
func (self *mockContainerHandler) GetStats() (*info.ContainerStats, error) {
args := self.Called()
return args.Get(0).(*info.ContainerStats), args.Error(1)
}
func (self *mockContainerHandler) ListContainers(listType ListType) ([]string, error) {
args := self.Called(listType)
return args.Get(0).([]string), args.Error(1)
}
func (self *mockContainerHandler) ListThreads(listType ListType) ([]int, error) {
args := self.Called(listType)
return args.Get(0).([]int), args.Error(1)
}
func (self *mockContainerHandler) ListProcesses(listType ListType) ([]int, error) {
args := self.Called(listType)
return args.Get(0).([]int), args.Error(1)
}
func TestWhiteListContainerFilter(t *testing.T) {
mockc := &mockContainerHandler{}
mockc.On("ListContainers", LIST_RECURSIVE).Return(
[]string{
"/docker/ee0103",
"/container/created/by/lmctfy",
"/user/something",
},
nil,
)
filterPaths := []string{
"/docker",
"/container",
}
fc := NewWhiteListFilter(mockc, filterPaths...)
containers, err := fc.ListContainers(LIST_RECURSIVE)
if err != nil {
t.Fatal(err)
}
for _, c := range containers {
legal := false
for _, prefix := range filterPaths {
if strings.HasPrefix(c, prefix) {
legal = true
}
}
if !legal {
t.Errorf("%v is not in the white list", c)
}
}
mockc.AssertExpectations(t)
}
func TestBlackListContainerFilter(t *testing.T) {
mockc := &mockContainerHandler{}
mockc.On("ListContainers", LIST_RECURSIVE).Return(
[]string{
"/docker/ee0103",
"/container/created/by/lmctfy",
"/user/something",
},
nil,
)
filterPaths := []string{
"/docker",
"/container",
}
fc := NewBlackListFilter(mockc, filterPaths...)
containers, err := fc.ListContainers(LIST_RECURSIVE)
if err != nil {
t.Fatal(err)
}
for _, c := range containers {
legal := true
for _, prefix := range filterPaths {
if strings.HasPrefix(c, prefix) {
legal = false
}
}
if !legal {
t.Errorf("%v is in the black list", c)
}
}
mockc.AssertExpectations(t)
}

52
container/lmctfy/factory.go Normal file
View File

@ -0,0 +1,52 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package lmctfy
import (
"errors"
"log"
"os/exec"
"github.com/google/cadvisor/container"
)
func Register(paths ...string) error {
if _, err := exec.LookPath("lmctfy"); err != nil {
return errors.New("cannot find lmctfy")
}
f := container.AddStatsSummaryToFactory(&lmctfyFactory{})
for _, path := range paths {
log.Printf("register lmctfy under %v", path)
container.RegisterContainerHandlerFactory(path, f)
}
return nil
}
type lmctfyFactory struct {
}
func (self *lmctfyFactory) String() string {
return "lmctfy"
}
func (self *lmctfyFactory) NewContainerHandler(name string) (container.ContainerHandler, error) {
c, err := New(name)
if err != nil {
return nil, err
}
// XXX(dengnan): /user is created by ubuntu 14.04. Not sure if we should list it
handler := container.NewBlackListFilter(c, "/user")
return handler, nil
}

2103
container/lmctfy/lmctfy.pb.go Normal file
View File

File diff suppressed because it is too large Load Diff

195
container/lmctfy/lmctfy_container.go Normal file
View File

@ -0,0 +1,195 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// A container object
package lmctfy
import (
"fmt"
"os/exec"
"strings"
"syscall"
"time"
"code.google.com/p/goprotobuf/proto"
"github.com/google/cadvisor/container"
"github.com/google/cadvisor/info"
)
type lmctfyContainerHandler struct {
// Container name
Name string
container.NoStatsSummary
}
const (
lmctfyBinary = "lmctfy"
notFoundExitCode = 5
)
// Create a new
func New(name string) (container.ContainerHandler, error) {
el := &lmctfyContainerHandler{
Name: name,
}
return el, nil
}
func getExitCode(err error) int {
msg, ok := err.(*exec.ExitError)
if ok {
return msg.Sys().(syscall.WaitStatus).ExitStatus()
}
return -1
}
func protobufToContainerSpec(pspec *ContainerSpec) *info.ContainerSpec {
ret := new(info.ContainerSpec)
if pspec.GetCpu() != nil {
cpuspec := new(info.CpuSpec)
cpuspec.Limit = pspec.GetCpu().GetLimit()
cpuspec.MaxLimit = pspec.GetCpu().GetMaxLimit()
if pspec.GetCpu().GetMask() != nil {
cpuspec.Mask.Data = pspec.GetCpu().GetMask().GetData()
}
ret.Cpu = cpuspec
}
if pspec.GetMemory() != nil {
pmem := pspec.GetMemory()
memspec := new(info.MemorySpec)
memspec.Limit = uint64(pmem.GetLimit())
memspec.Reservation = uint64(pmem.GetReservation())
memspec.SwapLimit = uint64(pmem.GetSwapLimit())
ret.Memory = memspec
}
return ret
}
// Gets spec.
func (c *lmctfyContainerHandler) GetSpec() (*info.ContainerSpec, error) {
// Run lmctfy spec "container_name" and get spec.
// Ignore if the container was not found.
cmd := exec.Command(lmctfyBinary, "spec", string(c.Name))
data, err := cmd.Output()
if err != nil && getExitCode(err) != notFoundExitCode {
return nil, fmt.Errorf("unable to run command %v spec %v: %v", lmctfyBinary, c.Name, err)
}
// Parse output into a protobuf.
pspec := &ContainerSpec{}
err = proto.UnmarshalText(string(data), pspec)
if err != nil {
return nil, err
}
spec := protobufToContainerSpec(pspec)
return spec, nil
}
func protobufToMemoryData(pmd *MemoryStats_MemoryData, data *info.MemoryStatsMemoryData) {
if pmd == nil {
return
}
data.Pgfault = uint64(pmd.GetPgfault())
data.Pgmajfault = uint64(pmd.GetPgmajfault())
return
}
func protobufToContainerStats(pstats *ContainerStats) *info.ContainerStats {
ret := new(info.ContainerStats)
if pstats.GetCpu() != nil {
pcpu := pstats.GetCpu()
cpustats := new(info.CpuStats)
cpustats.Usage.Total = pcpu.GetUsage().GetTotal()
percpu := pcpu.GetUsage().GetPerCpu()
if len(percpu) > 0 {
cpustats.Usage.PerCpu = make([]uint64, len(percpu))
for i, p := range percpu {
cpustats.Usage.PerCpu[i] = uint64(p)
}
}
cpustats.Usage.User = uint64(pcpu.GetUsage().GetUser())
cpustats.Usage.System = uint64(pcpu.GetUsage().GetSystem())
cpustats.Load = pcpu.GetLoad()
ret.Cpu = cpustats
}
if pstats.GetMemory() != nil {
pmem := pstats.GetMemory()
memstats := new(info.MemoryStats)
memstats.Limit = uint64(pmem.GetLimit())
memstats.Usage = uint64(pmem.GetUsage())
protobufToMemoryData(pmem.GetContainerData(), &memstats.ContainerData)
protobufToMemoryData(pmem.GetHierarchicalData(), &memstats.HierarchicalData)
ret.Memory = memstats
}
return ret
}
// Gets full stats.
func (c *lmctfyContainerHandler) GetStats() (*info.ContainerStats, error) {
// Ignore if the container was not found.
cmd := exec.Command(lmctfyBinary, "stats", "full", string(c.Name))
data, err := cmd.Output()
if err != nil && getExitCode(err) != notFoundExitCode {
return nil, fmt.Errorf("unable to run command %v stats full %v: %v", lmctfyBinary, c.Name, err)
}
// Parse output into a protobuf.
pstats := &ContainerStats{}
err = proto.UnmarshalText(string(data), pstats)
if err != nil {
return nil, err
}
stats := protobufToContainerStats(pstats)
stats.Timestamp = time.Now()
return stats, nil
}
// Gets all subcontainers.
func (c *lmctfyContainerHandler) ListContainers(listType container.ListType) ([]string, error) {
// Prepare the arguments.
args := []string{"list", "containers", "-v"}
if listType == container.LIST_RECURSIVE {
args = append(args, "-r")
}
args = append(args, c.Name)
// Run the command.
cmd := exec.Command(lmctfyBinary, args...)
data, err := cmd.Output()
if err != nil && getExitCode(err) != notFoundExitCode {
return nil, err
}
// Parse lines as container names.
if len(data) == 0 {
return []string{}, nil
}
names := strings.Split(string(data), "\n")
containerNames := make([]string, 0, len(names))
for _, name := range names {
if len(name) != 0 {
containerNames = append(containerNames, name)
}
}
return containerNames, nil
}
// TODO(vmarmol): Implement
func (c *lmctfyContainerHandler) ListThreads(listType container.ListType) ([]int, error) {
return []int{}, nil
}
func (c *lmctfyContainerHandler) ListProcesses(listType container.ListType) ([]int, error) {
return []int{}, nil
}

273
container/lmctfy/virtual_host.pb.go Normal file
View File

@ -0,0 +1,273 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Code generated by protoc-gen-go.
// source: virtual_host.proto
// DO NOT EDIT!
package lmctfy
import proto "code.google.com/p/goprotobuf/proto"
import json "encoding/json"
import math "math"
// Reference proto, json, and math imports to suppress error if they are not otherwise used.
var _ = proto.Marshal
var _ = &json.SyntaxError{}
var _ = math.Inf
type Network_Bridge_Type int32
const (
Network_Bridge_ETH Network_Bridge_Type = 0
Network_Bridge_OVS Network_Bridge_Type = 1
)
var Network_Bridge_Type_name = map[int32]string{
0: "ETH",
1: "OVS",
}
var Network_Bridge_Type_value = map[string]int32{
"ETH": 0,
"OVS": 1,
}
func (x Network_Bridge_Type) Enum() *Network_Bridge_Type {
p := new(Network_Bridge_Type)
*p = x
return p
}
func (x Network_Bridge_Type) String() string {
return proto.EnumName(Network_Bridge_Type_name, int32(x))
}
func (x *Network_Bridge_Type) UnmarshalJSON(data []byte) error {
value, err := proto.UnmarshalJSONEnum(Network_Bridge_Type_value, data, "Network_Bridge_Type")
if err != nil {
return err
}
*x = Network_Bridge_Type(value)
return nil
}
type Network struct {
Interface *string `protobuf:"bytes,1,opt,name=interface" json:"interface,omitempty"`
Connection *Network_Connection `protobuf:"bytes,3,opt,name=connection" json:"connection,omitempty"`
VirtualIp *Network_VirtualIp `protobuf:"bytes,2,opt,name=virtual_ip" json:"virtual_ip,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Network) Reset() { *m = Network{} }
func (m *Network) String() string { return proto.CompactTextString(m) }
func (*Network) ProtoMessage() {}
func (m *Network) GetInterface() string {
if m != nil && m.Interface != nil {
return *m.Interface
}
return ""
}
func (m *Network) GetConnection() *Network_Connection {
if m != nil {
return m.Connection
}
return nil
}
func (m *Network) GetVirtualIp() *Network_VirtualIp {
if m != nil {
return m.VirtualIp
}
return nil
}
type Network_VethPair struct {
Outside *string `protobuf:"bytes,1,opt,name=outside" json:"outside,omitempty"`
Inside *string `protobuf:"bytes,2,opt,name=inside" json:"inside,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Network_VethPair) Reset() { *m = Network_VethPair{} }
func (m *Network_VethPair) String() string { return proto.CompactTextString(m) }
func (*Network_VethPair) ProtoMessage() {}
func (m *Network_VethPair) GetOutside() string {
if m != nil && m.Outside != nil {
return *m.Outside
}
return ""
}
func (m *Network_VethPair) GetInside() string {
if m != nil && m.Inside != nil {
return *m.Inside
}
return ""
}
type Network_Bridge struct {
Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"`
Type *Network_Bridge_Type `protobuf:"varint,2,opt,name=type,enum=containers.Network_Bridge_Type" json:"type,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Network_Bridge) Reset() { *m = Network_Bridge{} }
func (m *Network_Bridge) String() string { return proto.CompactTextString(m) }
func (*Network_Bridge) ProtoMessage() {}
func (m *Network_Bridge) GetName() string {
if m != nil && m.Name != nil {
return *m.Name
}
return ""
}
func (m *Network_Bridge) GetType() Network_Bridge_Type {
if m != nil && m.Type != nil {
return *m.Type
}
return Network_Bridge_ETH
}
type Network_Connection struct {
VethPair *Network_VethPair `protobuf:"bytes,1,opt,name=veth_pair" json:"veth_pair,omitempty"`
Bridge *Network_Bridge `protobuf:"bytes,2,opt,name=bridge" json:"bridge,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Network_Connection) Reset() { *m = Network_Connection{} }
func (m *Network_Connection) String() string { return proto.CompactTextString(m) }
func (*Network_Connection) ProtoMessage() {}
func (m *Network_Connection) GetVethPair() *Network_VethPair {
if m != nil {
return m.VethPair
}
return nil
}
func (m *Network_Connection) GetBridge() *Network_Bridge {
if m != nil {
return m.Bridge
}
return nil
}
type Network_VirtualIp struct {
Ip *string `protobuf:"bytes,1,opt,name=ip" json:"ip,omitempty"`
Netmask *string `protobuf:"bytes,2,opt,name=netmask" json:"netmask,omitempty"`
Gateway *string `protobuf:"bytes,3,opt,name=gateway" json:"gateway,omitempty"`
Mtu *int32 `protobuf:"varint,4,opt,name=mtu" json:"mtu,omitempty"`
IpForward *bool `protobuf:"varint,5,opt,name=ip_forward" json:"ip_forward,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Network_VirtualIp) Reset() { *m = Network_VirtualIp{} }
func (m *Network_VirtualIp) String() string { return proto.CompactTextString(m) }
func (*Network_VirtualIp) ProtoMessage() {}
func (m *Network_VirtualIp) GetIp() string {
if m != nil && m.Ip != nil {
return *m.Ip
}
return ""
}
func (m *Network_VirtualIp) GetNetmask() string {
if m != nil && m.Netmask != nil {
return *m.Netmask
}
return ""
}
func (m *Network_VirtualIp) GetGateway() string {
if m != nil && m.Gateway != nil {
return *m.Gateway
}
return ""
}
func (m *Network_VirtualIp) GetMtu() int32 {
if m != nil && m.Mtu != nil {
return *m.Mtu
}
return 0
}
func (m *Network_VirtualIp) GetIpForward() bool {
if m != nil && m.IpForward != nil {
return *m.IpForward
}
return false
}
type Mounts struct {
Mount []*Mounts_Mount `protobuf:"bytes,1,rep,name=mount" json:"mount,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Mounts) Reset() { *m = Mounts{} }
func (m *Mounts) String() string { return proto.CompactTextString(m) }
func (*Mounts) ProtoMessage() {}
func (m *Mounts) GetMount() []*Mounts_Mount {
if m != nil {
return m.Mount
}
return nil
}
type Mounts_Mount struct {
Source *string `protobuf:"bytes,1,opt,name=source" json:"source,omitempty"`
Target *string `protobuf:"bytes,2,opt,name=target" json:"target,omitempty"`
ReadOnly *bool `protobuf:"varint,3,opt,name=read_only" json:"read_only,omitempty"`
Private *bool `protobuf:"varint,4,opt,name=private" json:"private,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Mounts_Mount) Reset() { *m = Mounts_Mount{} }
func (m *Mounts_Mount) String() string { return proto.CompactTextString(m) }
func (*Mounts_Mount) ProtoMessage() {}
func (m *Mounts_Mount) GetSource() string {
if m != nil && m.Source != nil {
return *m.Source
}
return ""
}
func (m *Mounts_Mount) GetTarget() string {
if m != nil && m.Target != nil {
return *m.Target
}
return ""
}
func (m *Mounts_Mount) GetReadOnly() bool {
if m != nil && m.ReadOnly != nil {
return *m.ReadOnly
}
return false
}
func (m *Mounts_Mount) GetPrivate() bool {
if m != nil && m.Private != nil {
return *m.Private
}
return false
}
func init() {
proto.RegisterEnum("containers.Network_Bridge_Type", Network_Bridge_Type_name, Network_Bridge_Type_value)
}

101
container/samplermngr.go Normal file
View File

@ -0,0 +1,101 @@
package container
import (
"fmt"
"sync"
"time"
"github.com/google/cadvisor/info"
"github.com/google/cadvisor/sampling"
)
type samplerFactor interface {
String() string
NewSampler(*StatsParameter) (sampling.Sampler, error)
}
type samplerManager struct {
factoryMap map[string]samplerFactor
lock sync.RWMutex
}
func (self *samplerManager) Register(factory samplerFactor) {
self.lock.Lock()
defer self.lock.Unlock()
if self.factoryMap == nil {
self.factoryMap = make(map[string]samplerFactor, 3)
}
self.factoryMap[factory.String()] = factory
}
func (self *samplerManager) NewSampler(param *StatsParameter) (sampling.Sampler, error) {
self.lock.RLock()
defer self.lock.RUnlock()
if f, ok := self.factoryMap[param.Sampler]; ok {
return f.NewSampler(param)
}
return nil, fmt.Errorf("unknown sampler %v", param.Sampler)
}
var globalSamplerManager samplerManager
func NewSampler(param *StatsParameter) (sampling.Sampler, error) {
return globalSamplerManager.NewSampler(param)
}
type reservoirSamplerFactory struct {
}
func (self *reservoirSamplerFactory) String() string {
return "uniform"
}
func (self *reservoirSamplerFactory) NewSampler(param *StatsParameter) (sampling.Sampler, error) {
s := sampling.NewReservoirSampler(param.NumSamples)
if param.ResetPeriod.Seconds() > 1.0 {
s = sampling.NewPeriodcallyResetSampler(param.ResetPeriod, s)
}
return s, nil
}
type esSamplerFactory struct {
startTime time.Time
}
func (self *esSamplerFactory) String() string {
return "weighted"
}
func (self *esSamplerFactory) NewSampler(param *StatsParameter) (sampling.Sampler, error) {
s := sampling.NewESSampler(param.NumSamples, func(d interface{}) float64 {
stats := d.(*info.ContainerStats)
delta := self.startTime.Sub(stats.Timestamp)
return delta.Seconds()
})
if param.ResetPeriod.Seconds() > 1.0 {
s = sampling.NewPeriodcallyResetSampler(param.ResetPeriod, s)
}
return s, nil
}
type chainSamplerFactory struct {
}
func (self *chainSamplerFactory) String() string {
return "window"
}
func (self *chainSamplerFactory) NewSampler(param *StatsParameter) (sampling.Sampler, error) {
s := sampling.NewChainSampler(param.NumSamples, param.WindowSize)
if param.ResetPeriod.Seconds() > 1.0 {
s = sampling.NewPeriodcallyResetSampler(param.ResetPeriod, s)
}
return s, nil
}
func init() {
globalSamplerManager.Register(&reservoirSamplerFactory{})
globalSamplerManager.Register(&esSamplerFactory{time.Now()})
globalSamplerManager.Register(&chainSamplerFactory{})
}

120
container/statssum.go Normal file
View File

@ -0,0 +1,120 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package container
import (
"math/big"
"sync"
"time"
"github.com/google/cadvisor/info"
"github.com/google/cadvisor/sampling"
)
type statsSummaryContainerHandlerWrapper struct {
handler ContainerHandler
currentSummary *info.ContainerStatsSummary
totalMemoryUsage *big.Int
numStats uint64
sampler sampling.Sampler
lock sync.Mutex
}
func (self *statsSummaryContainerHandlerWrapper) GetSpec() (*info.ContainerSpec, error) {
return self.handler.GetSpec()
}
func (self *statsSummaryContainerHandlerWrapper) GetStats() (*info.ContainerStats, error) {
stats, err := self.handler.GetStats()
if err != nil {
return nil, err
}
if stats == nil {
return nil, nil
}
stats.Timestamp = time.Now()
self.lock.Lock()
defer self.lock.Unlock()
self.sampler.Update(stats)
if self.currentSummary == nil {
self.currentSummary = new(info.ContainerStatsSummary)
}
self.numStats++
if stats.Memory != nil {
if stats.Memory.Usage > self.currentSummary.MaxMemoryUsage {
self.currentSummary.MaxMemoryUsage = stats.Memory.Usage
}
// XXX(dengnan): Very inefficient!
if self.totalMemoryUsage == nil {
self.totalMemoryUsage = new(big.Int)
}
usage := (&big.Int{}).SetUint64(stats.Memory.Usage)
self.totalMemoryUsage = self.totalMemoryUsage.Add(self.totalMemoryUsage, usage)
n := (&big.Int{}).SetUint64(self.numStats)
avg := (&big.Int{}).Div(self.totalMemoryUsage, n)
self.currentSummary.AvgMemoryUsage = avg.Uint64()
}
return stats, nil
}
func (self *statsSummaryContainerHandlerWrapper) ListContainers(listType ListType) ([]string, error) {
return self.handler.ListContainers(listType)
}
func (self *statsSummaryContainerHandlerWrapper) ListThreads(listType ListType) ([]int, error) {
return self.handler.ListThreads(listType)
}
func (self *statsSummaryContainerHandlerWrapper) ListProcesses(listType ListType) ([]int, error) {
return self.handler.ListProcesses(listType)
}
func (self *statsSummaryContainerHandlerWrapper) StatsSummary() (*info.ContainerStatsSummary, error) {
self.lock.Lock()
defer self.lock.Unlock()
samples := make([]*info.ContainerStats, 0, self.sampler.Len())
self.sampler.Map(func(d interface{}) {
stats := d.(*info.ContainerStats)
samples = append(samples, stats)
})
self.currentSummary.Samples = samples
// XXX(dengnan): propabily add to StatsParameter?
self.currentSummary.FillPercentiles(
[]int{50, 80, 90, 95, 99},
[]int{50, 80, 90, 95, 99},
)
return self.currentSummary, nil
}
type StatsParameter struct {
Sampler string
NumSamples int
WindowSize int
ResetPeriod time.Duration
}
func AddStatsSummary(handler ContainerHandler, parameter *StatsParameter) (ContainerHandler, error) {
sampler, err := NewSampler(parameter)
if err != nil {
return nil, err
}
return &statsSummaryContainerHandlerWrapper{
handler: handler,
currentSummary: &info.ContainerStatsSummary{},
sampler: sampler,
}, nil
}

98
container/statssum_test.go Normal file
View File

@ -0,0 +1,98 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package container
import (
crand "crypto/rand"
"encoding/binary"
"math/rand"
"testing"
"github.com/google/cadvisor/info"
)
func init() {
// NOTE(dengnan): Even if we picked a good random seed,
// the random number from math/rand is still not cryptographically secure!
var seed int64
binary.Read(crand.Reader, binary.LittleEndian, &seed)
rand.Seed(seed)
}
type randomStatsContainer struct {
NoStatsSummary
}
func (self *randomStatsContainer) GetSpec() (*info.ContainerSpec, error) {
return nil, nil
}
func (self *randomStatsContainer) GetStats() (*info.ContainerStats, error) {
stats := new(info.ContainerStats)
stats.Cpu = new(info.CpuStats)
stats.Memory = new(info.MemoryStats)
stats.Memory.Usage = uint64(rand.Intn(2048))
return stats, nil
}
func (self *randomStatsContainer) ListContainers(listType ListType) ([]string, error) {
return nil, nil
}
func (self *randomStatsContainer) ListThreads(listType ListType) ([]int, error) {
return nil, nil
}
func (self *randomStatsContainer) ListProcesses(listType ListType) ([]int, error) {
return nil, nil
}
func TestAvgMaxMemoryUsage(t *testing.T) {
handler, err := AddStatsSummary(
&randomStatsContainer{},
&StatsParameter{
Sampler: "uniform",
NumSamples: 10,
},
)
if err != nil {
t.Error(err)
}
var maxUsage uint64
var totalUsage uint64
N := 100
for i := 0; i < N; i++ {
stats, err := handler.GetStats()
if err != nil {
t.Errorf("Error when get stats: %v", err)
continue
}
if stats.Memory.Usage > maxUsage {
maxUsage = stats.Memory.Usage
}
totalUsage += stats.Memory.Usage
}
summary, err := handler.StatsSummary()
if err != nil {
t.Fatalf("Error when get summary: %v", err)
}
if summary.MaxMemoryUsage != maxUsage {
t.Fatalf("Max memory usage should be %v; received %v", maxUsage, summary.MaxMemoryUsage)
}
avg := totalUsage / uint64(N)
if summary.AvgMemoryUsage != avg {
t.Fatalf("Avg memory usage should be %v; received %v", avg, summary.AvgMemoryUsage)
}
}

48
container/statssumfactory.go Normal file
View File

@ -0,0 +1,48 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package container
import "fmt"
type statsSummaryFactory struct {
factory ContainerHandlerFactory
}
func (self *statsSummaryFactory) String() string {
return fmt.Sprintf("%v/stats", self.factory)
}
var globalStatsParameter StatsParameter
func (self *statsSummaryFactory) NewContainerHandler(name string) (ContainerHandler, error) {
h, err := self.factory.NewContainerHandler(name)
if err != nil {
return nil, err
}
return AddStatsSummary(h, &globalStatsParameter)
}
// This is a decorator for container factory. If the container handler created
// by a container factory does not implement stats summary method, then the factory
// could be decorated with this structure.
func AddStatsSummaryToFactory(factory ContainerHandlerFactory) ContainerHandlerFactory {
return &statsSummaryFactory{
factory: factory,
}
}
func SetStatsParameter(param *StatsParameter) {
globalStatsParameter = *param
}

231
info/container.go Normal file
View File

@ -0,0 +1,231 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package info
import (
"sort"
"time"
)
type CpuSpecMask struct {
Data []uint64 `json:"data,omitempty"`
}
type CpuSpec struct {
Limit uint64 `json:"limit"`
MaxLimit uint64 `json:"max_limit"`
Mask CpuSpecMask `json:"mask,omitempty"`
}
type MemorySpec struct {
// The amount of memory requested. Default is unlimited (-1).
// Units: bytes.
Limit uint64 `json:"limit,omitempty"`
// The amount of guaranteed memory. Default is 0.
// Units: bytes.
Reservation uint64 `json:"reservation,omitempty"`
// The amount of swap space requested. Default is unlimited (-1).
// Units: bytes.
SwapLimit uint64 `json:"swap_limit,omitempty"`
}
type ContainerSpec struct {
Cpu *CpuSpec `json:"cpu,omitempty"`
Memory *MemorySpec `json:"memory,omitempty"`
}
type ContainerInfo struct {
// The absolute name of the container.
Name string `json:"name"`
// The direct subcontainers of the current container.
Subcontainers []string `json:"subcontainers,omitempty"`
// The isolation used in the container.
Spec *ContainerSpec `json:"spec,omitempty"`
// Historical statistics gathered from the container.
Stats []*ContainerStats `json:"stats,omitempty"`
StatsSummary *ContainerStatsSummary `json:"stats_summary,omitempty"`
}
func (self *ContainerInfo) StatsAfter(ref time.Time) []*ContainerStats {
n := len(self.Stats) + 1
for i, s := range self.Stats {
if s.Timestamp.After(ref) {
n = i
break
}
}
if n > len(self.Stats) {
return nil
}
return self.Stats[n:]
}
func (self *ContainerInfo) StatsStartTime() time.Time {
var ret time.Time
for _, s := range self.Stats {
if s.Timestamp.Before(ret) || ret.IsZero() {
ret = s.Timestamp
}
}
return ret
}
func (self *ContainerInfo) StatsEndTime() time.Time {
var ret time.Time
for i := len(self.Stats) - 1; i >= 0; i-- {
s := self.Stats[i]
if s.Timestamp.After(ret) {
ret = s.Timestamp
}
}
return ret
}
type CpuStats struct {
Usage struct {
Total uint64 `json:"total"`
PerCpu []uint64 `json:"per_cpu,omitempty"`
User uint64 `json:"user"`
System uint64 `json:"system"`
} `json:"usage"`
Load int32 `json:"load"`
}
type MemoryStats struct {
// Memory limit, equivalent to "limit" in MemorySpec.
// Units: Bytes.
Limit uint64 `json:"limit,omitempty"`
// Usage statistics.
// Current memory usage, this includes all memory regardless of when it was
// accessed.
// Units: Bytes.
Usage uint64 `json:"usage,omitempty"`
// The amount of working set memory, this includes recently accessed memory,
// dirty memory, and kernel memmory. Working set is <= "usage".
// Units: Bytes.
WorkingSet uint64 `json:"working_set,omitempty"`
ContainerData MemoryStatsMemoryData `json:"container_data,omitempty"`
HierarchicalData MemoryStatsMemoryData `json:"hierarchical_data,omitempty"`
}
type MemoryStatsMemoryData struct {
Pgfault uint64 `json:"pgfault,omitempty"`
Pgmajfault uint64 `json:"pgmajfault,omitempty"`
}
type ContainerStats struct {
// The time of this stat point.
Timestamp time.Time `json:"timestamp"`
Cpu *CpuStats `json:"cpu,omitempty"`
Memory *MemoryStats `json:"memory,omitempty"`
}
// This is not exported.
// Use FillPercentile to calculate percentiles
type percentile struct {
Percentage int `json:"percentage"`
Value uint64 `json:"value"`
}
type ContainerStatsSummary struct {
// TODO(dengnan): More things?
MaxMemoryUsage uint64 `json:"max_memory_usage,omitempty"`
AvgMemoryUsage uint64 `json:"avg_memory_usage,omitempty"`
Samples []*ContainerStats `json:"samples,omitempty"`
MemoryUsagePercentiles []percentile `json:"memory_usage_percentiles,omitempty"`
CpuUsagePercentiles []percentile `json:"cpu_usage_percentiles,omitempty"`
}
type uint64Slice []uint64
func (self uint64Slice) Len() int {
return len(self)
}
func (self uint64Slice) Less(i, j int) bool {
return self[i] < self[j]
}
func (self uint64Slice) Swap(i, j int) {
self[i], self[j] = self[j], self[i]
}
func (self uint64Slice) Percentiles(ps ...int) []uint64 {
if len(self) == 0 {
return nil
}
ret := make([]uint64, 0, len(ps))
sort.Sort(self)
for _, p := range ps {
idx := (float64(p) / 100.0) * float64(len(self)+1)
if idx > float64(len(self)-1) {
ret = append(ret, self[len(self)-1])
} else {
ret = append(ret, self[int(idx)])
}
}
return ret
}
// len(bs) <= len(as)
func float64Zipuint64(as []int, bs []uint64) []percentile {
if len(bs) == 0 {
return nil
}
ret := make([]percentile, len(bs))
for i, b := range bs {
a := as[i]
ret[i] = percentile{
Percentage: a,
Value: b,
}
}
return ret
}
func (self *ContainerStatsSummary) FillPercentiles(cpuPercentages, memoryPercentages []int) {
if len(self.Samples) == 0 {
return
}
cpuUsages := make([]uint64, 0, len(self.Samples))
memUsages := make([]uint64, 0, len(self.Samples))
for _, sample := range self.Samples {
if sample == nil {
continue
}
if sample.Cpu != nil {
cpuUsages = append(cpuUsages, sample.Cpu.Usage.Total)
}
if sample.Memory != nil {
memUsages = append(memUsages, sample.Memory.Usage)
}
}
cpuPercentiles := uint64Slice(cpuUsages).Percentiles(cpuPercentages...)
memPercentiles := uint64Slice(memUsages).Percentiles(memoryPercentages...)
self.CpuUsagePercentiles = float64Zipuint64(cpuPercentages, cpuPercentiles)
self.MemoryUsagePercentiles = float64Zipuint64(memoryPercentages, memPercentiles)
}

86
info/container_test.go Normal file
View File

@ -0,0 +1,86 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package info
import (
"testing"
"time"
)
func TestStatsStartTime(t *testing.T) {
N := 10
stats := make([]*ContainerStats, 0, N)
ct := time.Now()
for i := 0; i < N; i++ {
s := &ContainerStats{
Timestamp: ct.Add(time.Duration(i) * time.Second),
}
stats = append(stats, s)
}
cinfo := &ContainerInfo{
Name: "/some/container",
Stats: stats,
}
ref := ct.Add(time.Duration(N-1) * time.Second)
end := cinfo.StatsEndTime()
if !ref.Equal(end) {
t.Errorf("end time is %v; should be %v", end, ref)
}
}
func TestStatsEndTime(t *testing.T) {
N := 10
stats := make([]*ContainerStats, 0, N)
ct := time.Now()
for i := 0; i < N; i++ {
s := &ContainerStats{
Timestamp: ct.Add(time.Duration(i) * time.Second),
}
stats = append(stats, s)
}
cinfo := &ContainerInfo{
Name: "/some/container",
Stats: stats,
}
ref := ct
start := cinfo.StatsStartTime()
if !ref.Equal(start) {
t.Errorf("start time is %v; should be %v", start, ref)
}
}
func TestPercentiles(t *testing.T) {
N := 100
data := make([]uint64, N)
for i := 0; i < N; i++ {
data[i] = uint64(i)
}
ps := []float64{
0.8,
0.9,
0.5,
}
ss := uint64Slice(data).Percentiles(ps...)
for i, s := range ss {
p := ps[i]
d := uint64(float64(N) * p)
if d != s {
t.Errorf("%v \\%tile data should be %v, but got %v", p*float64(100), d, s)
}
}
}

27
info/machine.go Normal file
View File

@ -0,0 +1,27 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package info
type MachineInfo struct {
// The number of cores in this machine.
NumCores int `json:"num_cores"`
// The amount of memory (in bytes) in this machine
MemoryCapacity int64 `json:"memory_capacity"`
}
type MachineInfoFactory interface {
GetMachineInfo() (*MachineInfo, error)
}

176
manager/container.go Normal file
View File

@ -0,0 +1,176 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Per-container manager.
package manager
import (
"container/list"
"flag"
"log"
"sync"
"time"
"github.com/google/cadvisor/container"
"github.com/google/cadvisor/info"
)
var historyDuration = flag.Int("history_duration", 60, "number of seconds of container history to keep")
// Internal mirror of the external data structure.
type containerStat struct {
Timestamp time.Time
Data *info.ContainerStats
}
type containerInfo struct {
Name string
Subcontainers []string
Spec *info.ContainerSpec
Stats *list.List
StatsSummary *info.ContainerStatsSummary
}
type containerData struct {
handler container.ContainerHandler
info containerInfo
lock sync.Mutex
// Tells the container to stop.
stop chan bool
}
func (c *containerData) Start() error {
// Force the first update.
c.housekeepingTick()
log.Printf("Start housekeeping for container %q\n", c.info.Name)
go c.housekeeping()
return nil
}
func (c *containerData) Stop() error {
c.stop <- true
return nil
}
func (c *containerData) GetInfo() (*containerInfo, error) {
// TODO(vmarmol): Consider caching this.
// Get spec and subcontainers.
err := c.updateSpec()
if err != nil {
return nil, err
}
err = c.updateSubcontainers()
if err != nil {
return nil, err
}
// Make a copy of the info for the user.
c.lock.Lock()
defer c.lock.Unlock()
ret := c.info
return &ret, nil
}
func NewContainerData(containerName string) (*containerData, error) {
cont := &containerData{}
handler, err := container.NewContainerHandler(containerName)
if err != nil {
return nil, err
}
cont.handler = handler
cont.info.Name = containerName
cont.info.Stats = list.New()
cont.stop = make(chan bool, 1)
return cont, nil
}
func (c *containerData) housekeeping() {
// Housekeep every second.
for true {
select {
case <-c.stop:
// Stop housekeeping when signaled.
return
case <-time.Tick(time.Second):
start := time.Now()
c.housekeepingTick()
// Log if housekeeping took longer than 120ms.
duration := time.Since(start)
if duration >= 120*time.Millisecond {
log.Printf("Housekeeping(%s) took %s", c.info.Name, duration)
}
}
}
}
func (c *containerData) housekeepingTick() {
err := c.updateStats()
if err != nil {
log.Printf("Failed to update stats for container \"%s\": %s", c.info.Name, err)
}
}
func (c *containerData) updateSpec() error {
spec, err := c.handler.GetSpec()
if err != nil {
return err
}
c.lock.Lock()
defer c.lock.Unlock()
c.info.Spec = spec
return nil
}
func (c *containerData) updateStats() error {
stats, err := c.handler.GetStats()
if err != nil {
return err
}
if stats == nil {
return nil
}
summary, err := c.handler.StatsSummary()
if err != nil {
return err
}
timestamp := time.Now()
// Remove the front if we go over.
c.lock.Lock()
defer c.lock.Unlock()
if c.info.Stats.Len() >= *historyDuration {
c.info.Stats.Remove(c.info.Stats.Front())
}
c.info.Stats.PushBack(&containerStat{
Timestamp: timestamp,
Data: stats,
})
c.info.StatsSummary = summary
return nil
}
func (c *containerData) updateSubcontainers() error {
subcontainers, err := c.handler.ListContainers(container.LIST_SELF)
if err != nil {
return err
}
c.lock.Lock()
defer c.lock.Unlock()
c.info.Subcontainers = subcontainers
return nil
}

61
manager/machine.go Normal file
View File

@ -0,0 +1,61 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package manager
import (
"fmt"
"io/ioutil"
"regexp"
"strconv"
"github.com/google/cadvisor/info"
)
var numCpuRegexp = regexp.MustCompile("processor\\t*: +[0-9]+")
var memoryCapacityRegexp = regexp.MustCompile("MemTotal: *([0-9]+) kB")
func getMachineInfo() (*info.MachineInfo, error) {
// Get the number of CPUs from /proc/cpuinfo.
out, err := ioutil.ReadFile("/proc/cpuinfo")
if err != nil {
return nil, err
}
numCores := len(numCpuRegexp.FindAll(out, -1))
if numCores == 0 {
return nil, fmt.Errorf("failed to count cores in output: %s", string(out))
}
// Get the amount of usable memory from /proc/meminfo.
out, err = ioutil.ReadFile("/proc/meminfo")
if err != nil {
return nil, err
}
matches := memoryCapacityRegexp.FindSubmatch(out)
if len(matches) != 2 {
return nil, fmt.Errorf("failed to find memory capacity in output: %s", string(out))
}
memoryCapacity, err := strconv.ParseInt(string(matches[1]), 10, 64)
if err != nil {
return nil, err
}
// Capacity is in KB, convert it to bytes.
memoryCapacity = memoryCapacity * 1024
return &info.MachineInfo{
NumCores: numCores,
MemoryCapacity: memoryCapacity,
}, nil
}

247
manager/manager.go Normal file
View File

@ -0,0 +1,247 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package manager
import (
"fmt"
"log"
"sync"
"time"
"github.com/google/cadvisor/container"
"github.com/google/cadvisor/info"
)
type Manager interface {
// Start the manager, blocks forever.
Start() error
// Get information about a container.
GetContainerInfo(containerName string) (*info.ContainerInfo, error)
// Get information about the machine.
GetMachineInfo() (*info.MachineInfo, error)
}
func New() (Manager, error) {
newManager := &manager{}
newManager.containers = make(map[string]*containerData)
machineInfo, err := getMachineInfo()
if err != nil {
return nil, err
}
newManager.machineInfo = *machineInfo
log.Printf("Machine: %+v", newManager.machineInfo)
return newManager, nil
}
type manager struct {
containers map[string]*containerData
containersLock sync.RWMutex
machineInfo info.MachineInfo
}
// Start the container manager.
func (m *manager) Start() error {
// Create root and then recover all containers.
_, err := m.createContainer("/")
if err != nil {
return err
}
log.Printf("Starting recovery of all containers")
err = m.detectContainers()
if err != nil {
return err
}
log.Printf("Recovery completed")
// Look for new containers in the main housekeeping thread.
for t := range time.Tick(time.Second) {
start := time.Now()
// Check for new containers.
err = m.detectContainers()
if err != nil {
log.Printf("Failed to detect containers: %s", err)
}
// Log if housekeeping took more than 100ms.
duration := time.Since(start)
if duration >= 100*time.Millisecond {
log.Printf("Global Housekeeping(%d) took %s", t.Unix(), duration)
}
}
return nil
}
// Get a container by name.
func (m *manager) GetContainerInfo(containerName string) (*info.ContainerInfo, error) {
log.Printf("Get(%s)", containerName)
var cont *containerData
var ok bool
func() {
m.containersLock.RLock()
defer m.containersLock.RUnlock()
// Ensure we have the container.
cont, ok = m.containers[containerName]
}()
if !ok {
return nil, fmt.Errorf("unknown container \"%s\"", containerName)
}
// Get the info from the container.
cinfo, err := cont.GetInfo()
if err != nil {
return nil, err
}
// Make a copy of the info for the user.
ret := &info.ContainerInfo{
Name: cinfo.Name,
Subcontainers: cinfo.Subcontainers,
Spec: cinfo.Spec,
StatsSummary: cinfo.StatsSummary,
}
// Set default value to an actual value
if ret.Spec.Memory != nil {
// Memory.Limit is 0 means there's no limit
if ret.Spec.Memory.Limit == 0 {
ret.Spec.Memory.Limit = uint64(m.machineInfo.MemoryCapacity)
}
}
ret.Stats = make([]*info.ContainerStats, 0, cinfo.Stats.Len())
for e := cinfo.Stats.Front(); e != nil; e = e.Next() {
data := e.Value.(*containerStat)
ret.Stats = append(ret.Stats, data.Data)
}
return ret, nil
}
func (m *manager) GetMachineInfo() (*info.MachineInfo, error) {
// Copy and return the MachineInfo.
ret := m.machineInfo
return &ret, nil
}
// Create a container. This expects to only be called from the global manager thread.
func (m *manager) createContainer(containerName string) (*containerData, error) {
cont, err := NewContainerData(containerName)
if err != nil {
return nil, err
}
// Add to the containers map.
func() {
m.containersLock.Lock()
defer m.containersLock.Unlock()
log.Printf("Added container: %s", containerName)
m.containers[containerName] = cont
}()
// Start the container's housekeeping.
cont.Start()
return cont, nil
}
func (m *manager) destroyContainer(containerName string) error {
m.containersLock.Lock()
defer m.containersLock.Unlock()
cont, ok := m.containers[containerName]
if !ok {
return fmt.Errorf("Expected container \"%s\" to exist during destroy", containerName)
}
// Tell the container to stop.
err := cont.Stop()
if err != nil {
return err
}
// Remove the container from our records.
delete(m.containers, containerName)
log.Printf("Destroyed container: %s", containerName)
return nil
}
type empty struct{}
// Detect all containers that have been added or deleted.
func (m *manager) getContainersDiff() (added []string, removed []string, err error) {
// TODO(vmarmol): We probably don't need to lock around / since it will always be there.
m.containersLock.RLock()
defer m.containersLock.RUnlock()
// Get all containers on the system.
cont, ok := m.containers["/"]
if !ok {
return nil, nil, fmt.Errorf("Failed to find container \"/\" while checking for new containers")
}
allContainers, err := cont.handler.ListContainers(container.LIST_RECURSIVE)
if err != nil {
return nil, nil, err
}
allContainers = append(allContainers, "/")
// Determine which were added and which were removed.
allContainersSet := make(map[string]*empty)
for name, _ := range m.containers {
allContainersSet[name] = &empty{}
}
for _, name := range allContainers {
delete(allContainersSet, name)
_, ok := m.containers[name]
if !ok {
added = append(added, name)
}
}
// Removed ones are no longer in the container listing.
for name, _ := range allContainersSet {
removed = append(removed, name)
}
return
}
// Detect the existing containers and reflect the setup here.
func (m *manager) detectContainers() error {
added, removed, err := m.getContainersDiff()
if err != nil {
return err
}
// Add the new containers.
for _, name := range added {
_, err = m.createContainer(name)
if err != nil {
return fmt.Errorf("Failed to create existing container: %s: %s", name, err)
}
}
// Remove the old containers.
for _, name := range removed {
err = m.destroyContainer(name)
if err != nil {
return fmt.Errorf("Failed to destroy existing container: %s: %s", name, err)
}
}
return nil
}

199
pages/containers.go Normal file
View File

@ -0,0 +1,199 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Page for /containers/
package pages
import (
"fmt"
"html/template"
"log"
"net/http"
"net/url"
"path"
"strconv"
"strings"
"time"
"github.com/google/cadvisor/info"
"github.com/google/cadvisor/manager"
)
const ContainersPage = "/containers/"
var funcMap = template.FuncMap{
"containerLink": containerLink,
"printMask": printMask,
"printCores": printCores,
"printMegabytes": printMegabytes,
"containerNameEquals": containerNameEquals,
"getMemoryUsage": getMemoryUsage,
"getMemoryUsagePercent": getMemoryUsagePercent,
"getHotMemoryPercent": getHotMemoryPercent,
"getColdMemoryPercent": getColdMemoryPercent,
}
// TODO(vmarmol): Consider housekeeping Spec too so we can show changes through time. We probably don't need it ever second though.
var pageTemplate *template.Template
type pageData struct {
ContainerName string
ParentContainers []string
Subcontainers []string
Spec *info.ContainerSpec
Stats []*info.ContainerStats
MachineInfo *info.MachineInfo
ResourcesAvailable bool
CpuAvailable bool
MemoryAvailable bool
}
func init() {
pageTemplate = template.New("containersTemplate").Funcs(funcMap)
_, err := pageTemplate.Parse(containersHtmlTemplate)
if err != nil {
log.Fatalf("Failed to parse template: %s", err)
}
}
// TODO(vmarmol): Escape this correctly.
func containerLink(containerName string, basenameOnly bool, cssClasses string) interface{} {
var displayName string
if basenameOnly {
displayName = path.Base(string(containerName))
} else {
displayName = string(containerName)
}
if containerName == "root" {
containerName = "/"
}
return template.HTML(fmt.Sprintf("<a class=\"%s\" href=\"%s%s\">%s</a>", cssClasses, ContainersPage[:len(ContainersPage)-1], containerName, displayName))
}
func containerNameEquals(c1 string, c2 string) bool {
return c1 == c2
}
func printMask(mask *info.CpuSpecMask, numCores int) interface{} {
// TODO(vmarmol): Detect this correctly.
// TODO(vmarmol): Support more than 64 cores.
rawMask := uint64(0)
if len(mask.Data) > 0 {
rawMask = mask.Data[0]
}
masks := make([]string, numCores)
for i := uint(0); i < uint(numCores); i++ {
coreClass := "inactive-cpu"
// by default, all cores are active
if ((0x1<<i)&rawMask) != 0 || len(mask.Data) == 0 {
coreClass = "active-cpu"
}
masks[i] = fmt.Sprintf("<span class=\"%s\">%d</span>", coreClass, i)
}
return template.HTML(strings.Join(masks, "&nbsp;"))
}
func printCores(millicores *uint64) string {
// TODO(vmarmol): Detect this correctly
if *millicores > 1024*1000 {
return "unlimited"
}
cores := float64(*millicores) / 1000
return strconv.FormatFloat(cores, 'f', 3, 64)
}
func toMegabytes(bytes uint64) float64 {
return float64(bytes) / (1 << 20)
}
func printMegabytes(bytes uint64) string {
// TODO(vmarmol): Detect this correctly
if bytes > (100 << 30) {
return "unlimited"
}
megabytes := toMegabytes(bytes)
return strconv.FormatFloat(megabytes, 'f', 3, 64)
}
func toMemoryPercent(usage uint64, spec *info.ContainerSpec) int {
return int((usage * 100) / (spec.Memory.Limit))
}
func getMemoryUsage(stats []*info.ContainerStats) string {
return strconv.FormatFloat(toMegabytes((stats[len(stats)-1].Memory.Usage)), 'f', 2, 64)
}
func getMemoryUsagePercent(spec *info.ContainerSpec, stats []*info.ContainerStats) int {
return toMemoryPercent((stats[len(stats)-1].Memory.Usage), spec)
}
func getHotMemoryPercent(spec *info.ContainerSpec, stats []*info.ContainerStats) int {
return toMemoryPercent((stats[len(stats)-1].Memory.WorkingSet), spec)
}
func getColdMemoryPercent(spec *info.ContainerSpec, stats []*info.ContainerStats) int {
latestStats := stats[len(stats)-1].Memory
return toMemoryPercent((latestStats.Usage)-(latestStats.WorkingSet), spec)
}
func ServerContainersPage(m manager.Manager, w http.ResponseWriter, u *url.URL) error {
start := time.Now()
// The container name is the path after the handler
containerName := u.Path[len(ContainersPage)-1:]
// Get the container.
cont, err := m.GetContainerInfo(containerName)
if err != nil {
return fmt.Errorf("Failed to get container \"%s\" with error: %s", containerName, err)
}
// Get the MachineInfo
machineInfo, err := m.GetMachineInfo()
if err != nil {
return err
}
// Make a list of the parent containers and their links
var parentContainers []string
parentContainers = append(parentContainers, string("root"))
parentName := ""
for _, part := range strings.Split(string(cont.Name), "/") {
if part == "" {
continue
}
parentName += "/" + part
parentContainers = append(parentContainers, string(parentName))
}
data := &pageData{
ContainerName: cont.Name,
ParentContainers: parentContainers,
Subcontainers: cont.Subcontainers,
Spec: cont.Spec,
Stats: cont.Stats,
MachineInfo: machineInfo,
ResourcesAvailable: cont.Spec.Cpu != nil || cont.Spec.Memory != nil,
CpuAvailable: cont.Spec.Cpu != nil,
MemoryAvailable: cont.Spec.Memory != nil,
}
err = pageTemplate.Execute(w, data)
if err != nil {
log.Printf("Failed to apply template: %s", err)
}
log.Printf("Request took %s", time.Since(start))
return nil
}

160
pages/containers_html.go Normal file
View File

@ -0,0 +1,160 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package pages
const containersHtmlTemplate = `
<html>
<head>
<title>cAdvisor - Container {{.ContainerName}}</title>
<!-- Latest compiled and minified CSS -->
<link rel="stylesheet" href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css">
<!-- Optional theme -->
<link rel="stylesheet" href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap-theme.min.css">
<link rel="stylesheet" href="/static/containers.css">
<!-- Latest compiled and minified JavaScript -->
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script>
<script type="text/javascript" src="https://www.google.com/jsapi"></script>
<script type="text/javascript" src="/static/containers.js"></script>
</head>
<body>
<div class="container theme-showcase" >
<div class="col-sm-12" id="logo">
</div>
<div class="col-sm-12">
<div class="page-header">
<h1>{{.ContainerName}}</h1>
</div>
<ol class="breadcrumb">
{{range $parentContainer := .ParentContainers}}
<li>{{containerLink $parentContainer true ""}}</li>
{{end}}
</ol>
</div>
{{if .Subcontainers}}
<div class="col-sm-12">
<div class="page-header">
<h3>Subcontainers</h3>
</div>
<div class="list-group">
{{range $subcontainer := .Subcontainers}}
{{containerLink $subcontainer false "list-group-item"}}
{{end}}
</div>
</div>
{{end}}
{{if .ResourcesAvailable}}
<div class="col-sm-12">
<div class="page-header">
<h3>Isolation</h3>
</div>
{{if .CpuAvailable}}
<ul class="list-group">
<li class="list-group-item active isolation-title panel-title">CPU</li>
{{if .Spec.Cpu.Limit}}
<li class="list-group-item"><span class="stat-label">Limit</span> {{printCores .Spec.Cpu.Limit}} <span class="unit-label">cores</span></li>
{{end}}
{{if .Spec.Cpu.MaxLimit}}
<li class="list-group-item"><span class="stat-label">Max Limit</span> {{printCores .Spec.Cpu.MaxLimit}} <span class="unit-label">cores</span></li>
{{end}}
{{if .Spec.Cpu.Mask}}
<li class="list-group-item"><span class="stat-label">Allowed Cores</span> {{printMask .Spec.Cpu.Mask .MachineInfo.NumCores}}</li>
{{end}}
</ul>
{{end}}
{{if .MemoryAvailable}}
<ul class="list-group">
<li class="list-group-item active isolation-title panel-title">Memory</li>
{{if .Spec.Memory.Reservation}}
<li class="list-group-item"><span class="stat-label">Reservation</span> {{printMegabytes .Spec.Memory.Reservation}} <span class="unit-label">MB</span></li>
{{end}}
{{if .Spec.Memory.Limit}}
<li class="list-group-item"><span class="stat-label">Limit</span> {{printMegabytes .Spec.Memory.Limit}} <span class="unit-label">MB</span></li>
{{end}}
{{if .Spec.Memory.SwapLimit}}
<li class="list-group-item"><span class="stat-label">Swap Limit</span> {{printMegabytes .Spec.Memory.SwapLimit}} <span class="unit-label">MB</span></li>
{{end}}
</ul>
{{end}}
</div>
<div class="col-sm-12">
<div class="page-header">
<h3>Usage</h3>
</div>
<div class="panel panel-primary">
<div class="panel-heading">
<h3 class="panel-title">Overview</h3>
</div>
<div id="usage-gauge" class="panel-body">
</div>
</div>
{{if .CpuAvailable}}
<div class="panel panel-primary">
<div class="panel-heading">
<h3 class="panel-title">CPU</h3>
</div>
<div class="panel-body">
<h4>Total Usage</h4>
<div id="cpu-total-usage-chart"></div>
<h4>Usage per Core</h4>
<div id="cpu-per-core-usage-chart"></div>
<h4>Usage Breakdown</h4>
<div id="cpu-usage-breakdown-chart"></div>
</div>
</div>
{{end}}
{{if .MemoryAvailable}}
<div class="panel panel-primary">
<div class="panel-heading">
<h3 class="panel-title">Memory</h3>
</div>
<div class="panel-body">
<h4>Total Usage</h4>
<div id="memory-usage-chart"></div>
<br/>
<div class="row col-sm-12">
<h4>Usage Breakdown</h4>
<div class="col-sm-9">
<div class="progress">
<div class="progress-bar progress-bar-danger" style="width: {{getHotMemoryPercent .Spec .Stats}}%">
<span class="sr-only">Hot Memory</span>
</div>
<div class="progress-bar progress-bar-info" style="width: {{getColdMemoryPercent .Spec .Stats}}%">
<span class="sr-only">Cold Memory</span>
</div>
</div>
</div>
<div class="col-sm-3">
{{ getMemoryUsage .Stats }} MB ({{ getMemoryUsagePercent .Spec .Stats }}%)
</div>
</div>
<h4>Page Faults</h4>
<div id="memory-page-faults-chart"></div>
</div>
</div>
{{end}}
</div>
{{end}}
</div>
<script type="text/javascript">
startPage({{.ContainerName}}, {{.CpuAvailable}}, {{.MemoryAvailable}});
</script>
</body>
</html>
`

47
pages/static/containers_css.go Normal file
View File

File diff suppressed because one or more lines are too long

288
pages/static/containers_js.go Normal file
View File

@ -0,0 +1,288 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package static
const containersJs = `
google.load("visualization", "1", {packages: ["corechart", "gauge"]});
// Draw a line chart.
function drawLineChart(seriesTitles, data, elementId, unit) {
// Convert the first column to a Date.
for (var i = 0; i < data.length; i++) {
if (data[i] != null) {
data[i][0] = new Date(data[i][0]);
}
}
// Add the definition of each column and the necessary data.
var dataTable = new google.visualization.DataTable();
dataTable.addColumn('datetime', seriesTitles[0]);
for (var i = 1; i < seriesTitles.length; i++) {
dataTable.addColumn('number', seriesTitles[i]);
}
dataTable.addRows(data);
// Create and draw the visualization.
var ac = null;
var opts = null;
// TODO(vmarmol): Remove this hack, it is to support the old charts and the new charts during the transition.
if (window.charts) {
if (!(elementId in window.charts)) {
ac = new google.visualization.LineChart(document.getElementById(elementId));
window.charts[elementId] = ac;
}
ac = window.charts[elementId];
opts = window.chartOptions;
} else {
ac = new google.visualization.LineChart(document.getElementById(elementId));
opts = {};
}
opts.vAxis = {title: unit};
ac.draw(dataTable, window.chartOptions);
}
// Draw a gauge.
function drawGauge(elementId, cpuUsage, memoryUsage) {
var gauges = [['Label', 'Value']];
if (cpuUsage >= 0) {
gauges.push(['CPU', cpuUsage]);
}
if (memoryUsage >= 0) {
gauges.push(['Memory', memoryUsage]);
}
// Create and populate the data table.
var data = google.visualization.arrayToDataTable(gauges);
// Create and draw the visualization.
var options = {
width: 400, height: 120,
redFrom: 90, redTo: 100,
yellowFrom:75, yellowTo: 90,
minorTicks: 5,
animation: {
duration: 900,
easing: 'linear'
}
};
var chart = new google.visualization.Gauge(document.getElementById(elementId));
chart.draw(data, options);
}
// Get the machine info.
function getMachineInfo(callback) {
$.getJSON("/api/v1.0/machine", function(data) {
callback(data);
});
}
// Get the container stats for the specified container.
function getStats(containerName, callback) {
$.getJSON("/api/v1.0/containers" + containerName, function(data) {
callback(data);
});
}
// Draw the graph for CPU usage.
function drawCpuTotalUsage(elementId, machineInfo, stats) {
var titles = ["Time", "Total"];
var data = [];
for (var i = 1; i < stats.stats.length; i++) {
var cur = stats.stats[i];
var prev = stats.stats[i - 1];
// TODO(vmarmol): This assumes we sample every second, use the timestamps.
var elements = [];
elements.push(cur.timestamp);
elements.push((cur.cpu.usage.total - prev.cpu.usage.total) / 1000000000);
data.push(elements);
}
drawLineChart(titles, data, elementId, "Cores");
}
// Draw the graph for per-core CPU usage.
function drawCpuPerCoreUsage(elementId, machineInfo, stats) {
// Add a title for each core.
var titles = ["Time"];
for (var i = 0; i < machineInfo.num_cores; i++) {
titles.push("Core " + i);
}
var data = [];
for (var i = 1; i < stats.stats.length; i++) {
var cur = stats.stats[i];
var prev = stats.stats[i - 1];
var elements = [];
elements.push(cur.timestamp);
for (var j = 0; j < machineInfo.num_cores; j++) {
// TODO(vmarmol): This assumes we sample every second, use the timestamps.
elements.push((cur.cpu.usage.per_cpu[j] - prev.cpu.usage.per_cpu[j]) / 1000000000);
}
data.push(elements);
}
drawLineChart(titles, data, elementId, "Cores");
}
// Draw the graph for CPU usage breakdown.
function drawCpuUsageBreakdown(elementId, containerInfo) {
var titles = ["Time", "User", "Kernel"];
var data = [];
for (var i = 1; i < containerInfo.stats.length; i++) {
var cur = containerInfo.stats[i];
var prev = containerInfo.stats[i - 1];
// TODO(vmarmol): This assumes we sample every second, use the timestamps.
var elements = [];
elements.push(cur.timestamp);
elements.push((cur.cpu.usage.user - prev.cpu.usage.user) / 1000000000);
elements.push((cur.cpu.usage.system - prev.cpu.usage.system) / 1000000000);
data.push(elements);
}
drawLineChart(titles, data, elementId, "Cores");
}
// Draw the gauges for overall resource usage.
function drawOverallUsage(elementId, containerInfo) {
var cur = containerInfo.stats[containerInfo.stats.length - 1];
var cpuUsage = 0;
if (containerInfo.spec.cpu && containerInfo.stats.length >= 2) {
var prev = containerInfo.stats[containerInfo.stats.length - 2];
var rawUsage = cur.cpu.usage.total - prev.cpu.usage.total;
// Convert to millicores and take the percentage
cpuUsage = Math.round(((rawUsage / 1000000) / containerInfo.spec.cpu.limit) * 100);
if (cpuUsage > 100) {
cpuUsage = 100;
}
}
var memoryUsage = 0;
if (containerInfo.spec.memory) {
memoryUsage = Math.round((cur.memory.usage / containerInfo.spec.memory.limit) * 100);
}
drawGauge(elementId, cpuUsage, memoryUsage);
}
var oneMegabyte = 1024 * 1024;
function drawMemoryUsage(elementId, containerInfo) {
var titles = ["Time", "Total"];
var data = [];
for (var i = 0; i < containerInfo.stats.length; i++) {
var cur = containerInfo.stats[i];
// TODO(vmarmol): This assumes we sample every second, use the timestamps.
var elements = [];
elements.push(cur.timestamp);
elements.push(cur.memory.usage / oneMegabyte);
data.push(elements);
}
drawLineChart(titles, data, elementId, "Megabytes");
}
function drawMemoryPageFaults(elementId, containerInfo) {
var titles = ["Time", "Faults", "Major Faults"];
var data = [];
for (var i = 1; i < containerInfo.stats.length; i++) {
var cur = containerInfo.stats[i];
var prev = containerInfo.stats[i - 1];
// TODO(vmarmol): This assumes we sample every second, use the timestamps.
var elements = [];
elements.push(cur.timestamp);
elements.push(cur.memory.hierarchical_data.pgfault - prev.memory.hierarchical_data.pgfault);
// TODO(vmarmol): Fix to expose this data.
//elements.push(cur.memory.hierarchical_data.pgmajfault - prev.memory.hierarchical_data.pgmajfault);
elements.push(0);
data.push(elements);
}
drawLineChart(titles, data, elementId, "Faults");
}
// Expects an array of closures to call. After each execution the JS runtime is given control back before continuing.
// This function returns asynchronously
function stepExecute(steps) {
// No steps, stop.
if (steps.length == 0) {
return;
}
// Get a step and execute it.
var step = steps.shift();
step();
// Schedule the next step.
setTimeout(function() {
stepExecute(steps);
}, 0);
}
// Draw all the charts on the page.
function drawCharts(machineInfo, containerInfo) {
var steps = [];
steps.push(function() {
drawOverallUsage("usage-gauge", containerInfo)
});
// CPU.
steps.push(function() {
drawCpuTotalUsage("cpu-total-usage-chart", machineInfo, containerInfo);
});
steps.push(function() {
drawCpuPerCoreUsage("cpu-per-core-usage-chart", machineInfo, containerInfo);
});
steps.push(function() {
drawCpuUsageBreakdown("cpu-usage-breakdown-chart", containerInfo);
});
// Memory.
steps.push(function() {
drawMemoryUsage("memory-usage-chart", containerInfo);
});
steps.push(function() {
drawMemoryPageFaults("memory-page-faults-chart", containerInfo);
});
stepExecute(steps);
}
// Executed when the page finishes loading.
function startPage(containerName, hasCpu, hasMemory) {
// Don't fetch data if we don't have any resource.
if (!hasCpu && !hasMemory) {
return;
}
// TODO(vmarmol): Look into changing the view window to get a smoother animation.
window.chartOptions = {
curveType: 'function',
height: 300,
legend:{position:"none"},
focusTarget: "category",
};
window.charts = {};
// Get machine info, then get the stats every 1s.
getMachineInfo(function(machineInfo) {
setInterval(function() {
getStats(containerName, function(stats){
drawCharts(machineInfo, stats);
});
}, 1000);
});
}
`

46
pages/static/static.go Normal file
View File

@ -0,0 +1,46 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Handler for /static content.
package static
import (
"fmt"
"net/http"
"net/url"
)
const StaticResource = "/static/"
var staticFiles = map[string]string{
"containers.css": containersCss,
"containers.js": containersJs,
}
func HandleRequest(w http.ResponseWriter, u *url.URL) error {
if len(u.Path) <= len(StaticResource) {
return fmt.Errorf("unknown static resource %q", u.Path)
}
// Get the static content if it exists.
resource := u.Path[len(StaticResource):]
content, ok := staticFiles[resource]
if !ok {
return fmt.Errorf("unknown static resource %q", resource)
}
_, err := w.Write([]byte(content))
return err
}

14
quickstart/Dockerfile Normal file
View File

@ -0,0 +1,14 @@
FROM ubuntu
MAINTAINER kyurtsever@google.com dengnan@google.com vmarmol@google.com
# Get the lmctfy dependencies.
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y --force-yes pkg-config libprotobuf8 libapparmor1
ADD http://storage.googleapis.com/cadvisor-bin/lmctfy/libre2.so.0.0.0 /usr/lib/libre2.so.0
# Get the lcmtfy and cAdvisor binaries.
ADD http://storage.googleapis.com/cadvisor-bin/lmctfy/lmctfy /usr/bin/lmctfy
ADD http://storage.googleapis.com/cadvisor-bin/cadvisor /usr/bin/cadvisor
RUN chmod +x /usr/bin/lmctfy && chmod +x /usr/bin/cadvisor
EXPOSE 8080

51
sampling/autofilter.go Normal file
View File

@ -0,0 +1,51 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sampling
type autoFilterSampler struct {
// filter will run to remove elements before adding every observation
filter func(d interface{}) bool
sampler Sampler
}
func (self *autoFilterSampler) Len() int {
return self.sampler.Len()
}
func (self *autoFilterSampler) Reset() {
self.sampler.Reset()
}
func (self *autoFilterSampler) Map(f func(d interface{})) {
self.sampler.Map(f)
}
func (self *autoFilterSampler) Filter(filter func(d interface{}) bool) {
self.sampler.Filter(filter)
}
func (self *autoFilterSampler) Update(d interface{}) {
self.Filter(self.filter)
self.sampler.Update(d)
}
// Add a decorator for sampler. Whenever an Update() is called, the sampler will
// call filter() first to remove elements in the decorated sampler.
func NewAutoFilterSampler(sampler Sampler, filter func(d interface{}) bool) Sampler {
return &autoFilterSampler{
filter: filter,
sampler: sampler,
}
}

60
sampling/autoreset.go Normal file
View File

@ -0,0 +1,60 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sampling
import "time"
type autoResetSampler struct {
shouldReset func(d interface{}) bool
sampler Sampler
}
func (self *autoResetSampler) Len() int {
return self.sampler.Len()
}
func (self *autoResetSampler) Reset() {
self.sampler.Reset()
}
func (self *autoResetSampler) Map(f func(d interface{})) {
self.sampler.Map(f)
}
func (self *autoResetSampler) Filter(filter func(d interface{}) bool) {
self.sampler.Filter(filter)
}
func (self *autoResetSampler) Update(d interface{}) {
if self.shouldReset(d) {
self.sampler.Reset()
}
self.sampler.Update(d)
}
func NewPeriodcallyResetSampler(period time.Duration, sampler Sampler) Sampler {
lastRest := time.Now()
shouldReset := func(d interface{}) bool {
if time.Now().Sub(lastRest) > period {
lastRest = time.Now()
return true
}
return false
}
return &autoResetSampler{
shouldReset: shouldReset,
sampler: sampler,
}
}

157
sampling/chainsample.go Normal file
View File

@ -0,0 +1,157 @@
package sampling
import (
"log"
"math/rand"
"sync"
"github.com/kr/pretty"
)
type empty struct{}
// Randomly generate number [start,end) except @except.
func randInt64Except(start, end int64, except map[int64]empty) int64 {
n := end - start
ret := rand.Int63n(n) + start
for _, ok := except[ret]; ok; _, ok = except[ret] {
ret = rand.Int63n(n) + start
}
return ret
}
// Basic idea:
// Every obervation will have a sequence number as its id.
// Suppose we want to sample k observations within latest n observations
// At first, we generated k random numbers in [0,n). These random numbers
// will be used as ids of observations that will be sampled.
type chainSampler struct {
sampleSize int
windowSize int64
// Every observation will have a sequence number starting from 1.
// The sequence number must increase by one for each observation.
numObservations int64
// All samples stored as id -> value.
samples map[int64]interface{}
// The set of id of future observations.
futureSamples map[int64]empty
// The chain of samples: old observation id -> future observation id.
// When the old observation expires, the future observation will be
// stored as a sample.
sampleChain map[int64]int64
// Replacements are: observations whose previous sample is not expired
// id->value.
replacements map[int64]interface{}
lock sync.RWMutex
}
func (self *chainSampler) initFutureSamples() {
for i := 0; i < self.sampleSize; i++ {
n := randInt64Except(1, self.windowSize+1, self.futureSamples)
self.futureSamples[n] = empty{}
}
}
func (self *chainSampler) arrive(seqNum int64, obv interface{}) {
if _, ok := self.futureSamples[seqNum]; !ok {
// If this observation is not selected, ignore it.
return
}
delete(self.futureSamples, seqNum)
if len(self.samples) < self.sampleSize {
self.samples[seqNum] = obv
}
self.replacements[seqNum] = obv
// Select a future observation which will replace current observation
// when it expires.
futureSeqNum := randInt64Except(seqNum+1, seqNum+self.windowSize+1, self.futureSamples)
self.futureSamples[futureSeqNum] = empty{}
self.sampleChain[seqNum] = futureSeqNum
}
func (self *chainSampler) expireAndReplace() {
expSeqNum := self.numObservations - self.windowSize
if _, ok := self.samples[expSeqNum]; !ok {
// No sample expires
return
}
delete(self.samples, expSeqNum)
// There must be a replacement, otherwise panic.
replacementSeqNum := self.sampleChain[expSeqNum]
// The sequence number must increase by one for each observation.
replacement, ok := self.replacements[replacementSeqNum]
if !ok {
log.Printf("cannot find %v. which is the replacement of %v\n", replacementSeqNum, expSeqNum)
pretty.Printf("chain: %# v\n", self)
panic("Should never occur!")
}
// This observation must have arrived before.
self.samples[replacementSeqNum] = replacement
}
func (self *chainSampler) Update(obv interface{}) {
self.lock.Lock()
defer self.lock.Unlock()
self.numObservations++
self.arrive(self.numObservations, obv)
self.expireAndReplace()
}
func (self *chainSampler) Len() int {
self.lock.RLock()
defer self.lock.RUnlock()
return len(self.samples)
}
func (self *chainSampler) Reset() {
self.lock.Lock()
defer self.lock.Unlock()
self.numObservations = 0
self.samples = make(map[int64]interface{}, self.sampleSize)
self.futureSamples = make(map[int64]empty, self.sampleSize*2)
self.sampleChain = make(map[int64]int64, self.sampleSize*2)
self.replacements = make(map[int64]interface{}, self.sampleSize*2)
self.initFutureSamples()
}
func (self *chainSampler) Map(f func(d interface{})) {
self.lock.RLock()
defer self.lock.RUnlock()
for seqNum, obv := range self.samples {
if _, ok := obv.(int); !ok {
pretty.Printf("Seq %v. WAT: %# v\n", seqNum, obv)
}
f(obv)
}
}
// NOT SUPPORTED
func (self *chainSampler) Filter(filter func(d interface{}) bool) {
return
}
// Chain sampler described in
// Brian Babcok, Mayur Datar and Rajeev Motwani,
// Sampling From a Moving Window Over Streaming Data
func NewChainSampler(sampleSize, windowSize int) Sampler {
sampler := &chainSampler{
sampleSize: sampleSize,
windowSize: int64(windowSize),
samples: make(map[int64]interface{}, sampleSize),
futureSamples: make(map[int64]empty, sampleSize*2),
sampleChain: make(map[int64]int64, sampleSize*2),
replacements: make(map[int64]interface{}, sampleSize*2),
}
sampler.initFutureSamples()
return sampler
}

29
sampling/chainsample_test.go Normal file
View File

@ -0,0 +1,29 @@
package sampling
import "testing"
func TestChainSampler(t *testing.T) {
numSamples := 10
windowSize := 10 * numSamples
numObservations := 10 * windowSize
numSampleRounds := 10 * numObservations
s := NewChainSampler(numSamples, windowSize)
hist := make(map[int]int, numSamples)
for i := 0; i < numSampleRounds; i++ {
sampleStream(hist, numObservations, s)
}
ratio := histStddev(hist) / histMean(hist)
if ratio > 1.05 {
// XXX(dengnan): better sampler?
t.Errorf("std dev: %v; mean: %v. Either we have a really bad PRNG, or a bad implementation", histStddev(hist), histMean(hist))
}
if len(hist) > windowSize {
t.Errorf("sampled %v data. larger than window size %v", len(hist), windowSize)
}
for seqNum, freq := range hist {
if seqNum < numObservations-windowSize && freq > 0 {
t.Errorf("observation with seqnum %v is sampled %v times", seqNum, freq)
}
}
}

17
sampling/doc.go Normal file
View File

@ -0,0 +1,17 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package sampling provides several sampling algorithms.
// These algorithms will be used to sample containers' stats information
package sampling

143
sampling/es.go Normal file
View File

@ -0,0 +1,143 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sampling
import (
"container/heap"
"math"
"math/rand"
"sync"
)
type esSampleItem struct {
data interface{}
key float64
}
type esSampleHeap []esSampleItem
func (self esSampleHeap) Len() int {
return len(self)
}
func (self esSampleHeap) Less(i, j int) bool {
return self[i].key < self[j].key
}
func (self esSampleHeap) Swap(i, j int) {
self[i], self[j] = self[j], self[i]
}
func (self *esSampleHeap) Push(x interface{}) {
item := x.(esSampleItem)
*self = append(*self, item)
}
func (self *esSampleHeap) Pop() interface{} {
old := *self
item := old[len(old)-1]
*self = old[:len(old)-1]
return item
}
type esSampler struct {
weight func(interface{}) float64
samples *esSampleHeap
maxSize int
lock sync.RWMutex
}
func (self *esSampler) Update(d interface{}) {
self.lock.Lock()
defer self.lock.Unlock()
u := rand.Float64()
key := math.Pow(u, 1.0/self.weight(d))
if self.samples.Len() < self.maxSize {
heap.Push(self.samples, esSampleItem{
data: d,
key: key,
})
return
}
s := *(self.samples)
min := s[0]
// The key of the new item is larger than a key in existing item.
// Add this new item.
if key > min.key {
heap.Pop(self.samples)
heap.Push(self.samples, esSampleItem{
data: d,
key: key,
})
}
}
func (self *esSampler) Len() int {
self.lock.RLock()
defer self.lock.RUnlock()
return len(*self.samples)
}
func (self *esSampler) Reset() {
self.lock.Lock()
defer self.lock.Unlock()
self.samples = &esSampleHeap{}
heap.Init(self.samples)
}
func (self *esSampler) Map(f func(interface{})) {
self.lock.RLock()
defer self.lock.RUnlock()
for _, d := range *self.samples {
f(d.data)
}
}
func (self *esSampler) Filter(filter func(d interface{}) bool) {
self.lock.Lock()
defer self.lock.Unlock()
rmlist := make([]int, 0, len(*self.samples))
for i, d := range *self.samples {
if filter(d.data) {
rmlist = append(rmlist, i)
}
}
for _, i := range rmlist {
heap.Remove(self.samples, i)
}
}
// ES sampling algorithm described in
//
// Pavlos S. Efraimidis and Paul G. Spirakis. Weighted random sampling with a
// reservoir. Information Processing Letters, 97(5):181 – 185, 2006.
//
// http://dl.acm.org/citation.cfm?id=1138834
func NewESSampler(size int, weight func(interface{}) float64) Sampler {
s := &esSampleHeap{}
heap.Init(s)
return &esSampler{
maxSize: size,
samples: s,
weight: weight,
}
}

81
sampling/es_test.go Normal file
View File

@ -0,0 +1,81 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sampling
import (
"container/heap"
"math/rand"
"testing"
"github.com/kr/pretty"
)
// This should be a min heap
func TestESSampleHeap(t *testing.T) {
h := &esSampleHeap{}
heap.Init(h)
min := 5.0
N := 10
for i := 0; i < N; i++ {
key := rand.Float64()
if key < min {
min = key
}
heap.Push(h, esSampleItem{nil, key})
}
l := *h
if l[0].key != min {
t.Errorf("not a min heap")
pretty.Printf("min=%v\nheap=%# v\n", min, l)
}
}
func TestESSampler(t *testing.T) {
reservoirSize := 10
numObvs := 10 * reservoirSize
numSampleRounds := 100 * numObvs
weight := func(d interface{}) float64 {
n := d.(int)
return float64(n + 1)
}
s := NewESSampler(reservoirSize, weight)
hist := make(map[int]int, numObvs)
for i := 0; i < numSampleRounds; i++ {
sampleStream(hist, numObvs, s)
}
diff := 2
wrongOrderedItems := make([]int, 0, numObvs)
threshold := 1.05
for i := 0; i < numObvs-diff; i++ {
// Item with smaller weight should have lower probability to be selected.
n1 := hist[i]
n2 := hist[i+diff]
if n1 > n2 {
if float64(n1) > float64(n2)*threshold {
wrongOrderedItems = append(wrongOrderedItems, i)
}
}
}
if float64(len(wrongOrderedItems)) > float64(numObvs)*0.05 {
for _, i := range wrongOrderedItems {
n1 := hist[i]
n2 := hist[i+diff]
t.Errorf("item with weight %v is selected %v times; while item with weight %v is selected %v times", i, n1, i+diff, n2)
}
}
}

99
sampling/reservoir.go Normal file
View File

@ -0,0 +1,99 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sampling
import (
"math/rand"
"sync"
)
// Reservoir sampling algorithm.
// http://en.wikipedia.org/wiki/Reservoir_sampling
type reservoirSampler struct {
maxSize int
samples []interface{}
numInstances int64
lock sync.RWMutex
}
func (self *reservoirSampler) Len() int {
self.lock.RLock()
defer self.lock.RUnlock()
return len(self.samples)
}
func (self *reservoirSampler) Reset() {
self.lock.Lock()
defer self.lock.Unlock()
self.samples = make([]interface{}, 0, self.maxSize)
self.numInstances = 0
}
// Update samples according to http://en.wikipedia.org/wiki/Reservoir_sampling
func (self *reservoirSampler) Update(d interface{}) {
self.lock.Lock()
defer self.lock.Unlock()
self.numInstances++
if len(self.samples) < self.maxSize {
self.samples = append(self.samples, d)
return
}
// Randomly generates a number between [0, numInances).
// Use this random number, j, as an index. If j is larger than the
// reservoir size, we will ignore the current new data.
// Otherwise replace the jth element in reservoir with the new data.
j := rand.Int63n(self.numInstances)
if j < int64(len(self.samples)) {
self.samples[int(j)] = d
}
}
func (self *reservoirSampler) Map(f func(d interface{})) {
self.lock.RLock()
defer self.lock.RUnlock()
for _, d := range self.samples {
f(d)
}
}
// Once an element is removed, the probability of sampling an observation will
// be increased. Removing all elements in the sampler has the same effect as
// calling Reset(). However, it will not guarantee the uniform probability of
// all unfiltered samples.
func (self *reservoirSampler) Filter(filter func(d interface{}) bool) {
self.lock.Lock()
defer self.lock.Unlock()
rmlist := make([]int, 0, len(self.samples))
for i, d := range self.samples {
if filter(d) {
rmlist = append(rmlist, i)
}
}
for _, i := range rmlist {
// slice trick: remove the ith element without preserving the order
self.samples[i] = self.samples[len(self.samples)-1]
self.samples = self.samples[:len(self.samples)-1]
}
self.numInstances -= int64(len(rmlist))
}
func NewReservoirSampler(reservoirSize int) Sampler {
return &reservoirSampler{
maxSize: reservoirSize,
}
}

70
sampling/reservoir_test.go Normal file
View File

@ -0,0 +1,70 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sampling
import (
"math"
"testing"
)
func sampleStream(hist map[int]int, n int, s Sampler) {
s.Reset()
for i := 0; i < n; i++ {
s.Update(i)
}
s.Map(func(d interface{}) {
j := d.(int)
if _, ok := hist[j]; !ok {
hist[j] = 0
}
hist[j]++
})
}
func histMean(hist map[int]int) float64 {
total := 0
for _, v := range hist {
total += v
}
return float64(total) / float64(len(hist))
}
func histStddev(hist map[int]int) float64 {
mean := histMean(hist)
var totalDiff float64
for _, v := range hist {
diff := float64(v) - mean
sq := diff * diff
totalDiff += sq
}
return math.Sqrt(totalDiff / float64(len(hist)))
}
// XXX(dengnan): This test may take more than 10 seconds.
func TestReservoirSampler(t *testing.T) {
reservoirSize := 10
numSamples := 10 * reservoirSize
numSampleRounds := 100 * numSamples
s := NewReservoirSampler(reservoirSize)
hist := make(map[int]int, numSamples)
for i := 0; i < numSampleRounds; i++ {
sampleStream(hist, numSamples, s)
}
ratio := histStddev(hist) / histMean(hist)
if ratio > 0.05 {
t.Errorf("std dev: %v; mean: %v. Either we have a really bad PRNG, or a bad implementation", histStddev(hist), histMean(hist))
}
}

42
sampling/sampler.go Normal file
View File

@ -0,0 +1,42 @@
// Copyright 2014 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package sampling
import (
crand "crypto/rand"
"encoding/binary"
"math/rand"
)
func init() {
// NOTE(dengnan): Even if we picked a good random seed,
// the random number from math/rand is still not cryptographically secure!
var seed int64
binary.Read(crand.Reader, binary.LittleEndian, &seed)
rand.Seed(seed)
}
type Sampler interface {
Update(d interface{})
Len() int
Reset()
Map(f func(interface{}))
// Filter() should update in place. Removing elements may or may not
// affect the statistical behavior of the sampler, i.e. the probability
// that an obervation will be sampled after removing some elements is
// implementation defined.
Filter(filter func(interface{}) bool)
}