go-字符串转成byte数组，会发生内存拷贝吗？

问题

字符串转成byte数组，会发生内存拷贝吗？

回答

字符串转成切片，会产生拷贝。严格来说，只要是发生类型强转都会发生内存拷贝。那么问题来了。

频繁的内存拷贝操作听起来对性能不大友好。有没有什么办法可以在字符串转成切片的时候不用发生拷贝呢？

解释

1
2
3
4
5
6
7
8
9
10
11
12
13
14
package main

import (
 "fmt"
 "reflect"
 "unsafe"
)

func main() {
 a :="aaa"
 ssh := *(*reflect.StringHeader)(unsafe.Pointer(&a))
 b := *(*[]byte)(unsafe.Pointer(&ssh))  
 fmt.Printf("%v",b)
}

StringHeader 是字符串在go的底层结构。

1
2
3
4
type StringHeader struct {
 Data uintptr
 Len  int
}

SliceHeader 是切片在go的底层结构。

1
2
3
4
5
type SliceHeader struct {
 Data uintptr
 Len  int
 Cap  int
}

那么如果想要在底层转换二者，只需要把 StringHeader 的地址强转成 SliceHeader 就行。那么go有个很强的包叫 unsafe 。

  1. unsafe.Pointer(&a)方法可以得到变量a的地址。
  2. (*reflect.StringHeader)(unsafe.Pointer(&a)) 可以把字符串a转成底层结构的形式。
  3. (*[]byte)(unsafe.Pointer(&ssh)) 可以把ssh底层结构体转成byte的切片的指针。
  4. 再通过 *转为指针指向的实际内容。

go-简单聊聊内存逃逸

问题

知道golang的内存逃逸吗？什么情况下会发生内存逃逸？

回答

golang程序变量会携带有一组校验数据，用来证明它的整个生命周期是否在运行时完全可知。如果变量通过了这些校验，它就可以在栈上分配。否则就说它 逃逸 了，必须在堆上分配。

能引起变量逃逸到堆上的典型情况：

  • 在方法内把局部变量指针返回 局部变量原本应该在栈中分配，在栈中回收。但是由于返回时被外部引用，因此其生命周期大于栈，则溢出。
  • 发送指针或带有指针的值到 channel 中。 在编译时，是没有办法知道哪个 goroutine 会在 channel 上接收数据。所以编译器没法知道变量什么时候才会被释放。
  • 在一个切片上存储指针或带指针的值。 一个典型的例子就是 []*string 。这会导致切片的内容逃逸。尽管其后面的数组可能是在栈上分配的，但其引用的值一定是在堆上。
  • slice 的背后数组被重新分配了，因为 append 时可能会超出其容量( cap )。 slice 初始化的地方在编译时是可以知道的，它最开始会在栈上分配。如果切片背后的存储要基于运行时的数据进行扩充，就会在堆上分配。
  • 在 interface 类型上调用方法。 在 interface 类型上调用方法都是动态调度的 —— 方法的真正实现只能在运行时知道。想像一个 io.Reader 类型的变量 r , 调用 r.Read(b) 会使得 r 的值和切片b 的背后存储都逃逸掉，所以会在堆上分配。

举例

通过一个例子加深理解，接下来尝试下怎么通过 go build -gcflags=-m 查看逃逸的情况。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
package main
import "fmt"
type A struct {
 s string
}
// 这是上面提到的 "在方法内把局部变量指针返回" 的情况
func foo(s string) *A {
 a := new(A) 
 a.s = s
 return a //返回局部变量a,在C语言中妥妥野指针，但在go则ok，但a会逃逸到堆
}
func main() {
 a := foo("hello")
 b := a.s + " world"
 c := b + "!"
 fmt.Println(c)
}

执行go build -gcflags=-m main.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
go build -gcflags=-m main.go
# command-line-arguments
./main.go:7:6: can inline foo
./main.go:13:10: inlining call to foo
./main.go:16:13: inlining call to fmt.Println
/var/folders/45/qx9lfw2s2zzgvhzg3mtzkwzc0000gn/T/go-build409982591/b001/_gomod_.go:6:6: can inline init.0
./main.go:7:10: leaking param: s
./main.go:8:10: new(A) escapes to heap
./main.go:16:13: io.Writer(os.Stdout) escapes to heap
./main.go:16:13: c escapes to heap
./main.go:15:9: b + "!" escapes to heap
./main.go:13:10: main new(A) does not escape
./main.go:14:11: main a.s + " world" does not escape
./main.go:16:13: main []interface {} literal does not escape
<autogenerated>:1: os.(*File).close .this does not escape
  • ./main.go:8:10: new(A) escapes to heap 说明 new(A) 逃逸了,符合上述提到的常见情况中的第一种。
  • ./main.go:14:11: main a.s + " world" does not escape 说明 b 变量没有逃逸，因为它只在方法内存在，会在方法结束时被回收。
  • ./main.go:15:9: b + "!" escapes to heap 说明 c 变量逃逸，通过fmt.Println(a ...interface{})打印的变量，都会发生逃逸，感兴趣的朋友可以去查查为什么。

以上操作其实就叫逃逸分析。下篇文章，跟大家聊聊怎么用一个比较trick的方法使变量不逃逸。方便大家在面试官面前秀一波。

openstack-一次虚拟机启动异常排查

环境 ： openstack stein版本，本地盘存储模式

故障：虚机启动异常，排查原因

查看虚机日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
2021-02-08 13:58:08.352 7 DEBUG nova.virt.images [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] 5516a827-a2fb-463b-9b03-168e56faa79f was qcow2, converting to raw fetch_to_raw /var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/images.py:172
2021-02-08 13:58:08.353 7 DEBUG nova.privsep.utils [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] Path '/var/lib/nova/instances' supports direct I/O supports_direct_io /var/lib/kolla/venv/lib/python2.7/site-packages/nova/privsep/utils.py:66
2021-02-08 13:58:08.353 7 DEBUG oslo_concurrency.processutils [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] Running cmd (subprocess): qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instances/_base/344d45fde44ae9163acfa37ab64364ffe2fd0b9c.part /var/lib/nova/instances/_base/344d45fde44ae9163acfa37ab64364ffe2fd0b9c.converted execute /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372
2021-02-08 13:58:10.109 7 DEBUG oslo_concurrency.processutils [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] CMD "qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instances/_base/344d45fde44ae9163acfa37ab64364ffe2fd0b9c.part /var/lib/nova/instances/_base/344d45fde44ae9163acfa37ab64364ffe2fd0b9c.converted" returned: 0 in 1.755s execute /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409
2021-02-08 13:58:10.375 7 DEBUG oslo_concurrency.processutils [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] Running cmd (subprocess): /var/lib/kolla/venv/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/_base/344d45fde44ae9163acfa37ab64364ffe2fd0b9c.converted --force-share execute /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372
2021-02-08 13:58:10.813 7 DEBUG oslo_concurrency.processutils [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] CMD "/var/lib/kolla/venv/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/_base/344d45fde44ae9163acfa37ab64364ffe2fd0b9c.converted --force-share" returned: 0 in 0.438s execute /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409
2021-02-08 13:58:10.819 7 DEBUG oslo_concurrency.lockutils [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] Lock "344d45fde44ae9163acfa37ab64364ffe2fd0b9c" released by "nova.virt.libvirt.imagebackend.fetch_func_sync" :: held 46.934s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757] Instance failed to spawn: ImageUnacceptable: Image 5516a827-a2fb-463b-9b03-168e56faa79f is unacceptable: Converted to raw, but format is now qcow2
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757] Traceback (most recent call last):
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2613, in _build_resources
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     yield resources
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2372, in _build_and_run_instance
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     block_device_info=block_device_info)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3358, in spawn
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     block_device_info=block_device_info)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3754, in _create_image
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     fallback_from_host)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3855, in _create_and_inject_local_root
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     instance, size, fallback_from_host)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8578, in _try_fetch_image_cache
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     trusted_certs=instance.trusted_certs)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 276, in cache
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     *args, **kwargs)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 668, in create_image
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     prepare_template(target=base, *args, **kwargs)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     return f(*args, **kwargs)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 272, in fetch_func_sync
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     fetch_func(target=target, *args, **kwargs)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 421, in fetch_image
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     images.fetch_to_raw(context, image_id, target, trusted_certs)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/images.py", line 188, in fetch_to_raw
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]     data.file_format)
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757] ImageUnacceptable: Image 5516a827-a2fb-463b-9b03-168e56faa79f is unacceptable: Converted to raw, but format is now qcow2
2021-02-08 13:58:10.820 7 ERROR nova.compute.manager [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757]
2021-02-08 13:58:10.821 7 INFO nova.compute.manager [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757] Terminating instance
2021-02-08 13:58:10.823 7 DEBUG nova.compute.manager [req-dd576230-3c75-41ca-b9f3-08b9c957eac5 1c935a2c05c140e18d93f3e189fd0ad7 3a228708dc8742d1adf519b52b613acf - b0a8dae41bc44b3296ae1a75d3e0c370 b0a8dae41bc44b3296ae1a75d3e0c370] [instance: bc2b194c-d28c-4bd1-960f-42950e5ab757] Start destroying the instance on the hypervisor. _shutdown_instance /var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py:2751

pycharm 查看nova代码

Read More

go-练习题

  1. 交替打印数字和字母

    使用两个 goroutine 交替打印序列，一个 goroutine 打印数字， 另外一个 goroutine 打印字母， 最终效果如下：

    1
    12AB34CD56EF78GH910IJ1112KL1314MN1516OP1718QR1920ST2122UV2324WX2526YZ2728

    思路：

    代码：

    1

  1. 判断字符串中字符是否全都不同

    请实现一个算法，确定一个字符串的所有字符【是否全都不同】。这里我们要求【不允许使用额外的存储结构】。 给定一个string，请返回一个bool值,true代表所有字符全都不同，false代表存在相同的字符。 保证字符串中的字符为【ASCII字符】。字符串的长度小于等于【3000】。

    思路:

    代码：

    1

openstack-gnocchi metricd报错

This is the error message in /var/lib/docker/volumes/kolla_logs/_data/gnocchi/gnocchi-metricd.log:

2018-07-05 08:12:36,365 [9755] ERROR cotyledon.utils: Unhandled exception
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/cotyledon/utils.py”, line 84, in exit_on_exception
yield
File “/usr/lib/python2.7/site-packages/cotyledon/service.py”, line 139, in _run
self.run()
File “/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py”, line 76, in run
self._configure()
File “/usr/lib/python2.7/site-packages/tenacity/__init.py”, line 214, in wrapped_f
return self.call(f, *args, **kw)
File “/usr/lib/python2.7/site-packages/tenacity/__init_.py”, line 295, in call
start_time=start_time)
File “/usr/lib/python2.7/site-packages/tenacity/init.py”, line 252, in iter
return fut.result()
File “/usr/lib/python2.7/site-packages/concurrent/futures/base.py”, line 422, in result
return self.__get_result()
File “/usr/lib/python2.7/site-packages/tenacity/__init_.py”, line 298, in call
result = fn(*args, **kwargs)
File “/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py”, line 161, in configure
six.moves.range(self.incoming.NUM_SACKS))
File “/usr/lib/python2.7/site-packages/gnocchi/incoming/__init_.py”, line 53, in NUM_SACKS
raise SackDetectionError(e)
SackDetectionError: [Errno 2] No such file or directory: ‘/var/lib/gnocchi/tmp/gnocchi-config’

解决方法：

1
2
#gnocchi-upgrade

SPF漏掉之DNS记录设置

漏洞描述

SPF 记录是一种域名服务（DNS）记录，用于标识哪些邮件服务器可以代表您的域名发送电子邮件。 SPF 记录的目的是为了防止垃圾邮件发送者在您的域名上，使用伪造的发件人地址发送邮件。

若您未对您的域名添加 SPF 解析记录，则黑客可以仿冒以该域名为后缀的邮箱，来发送垃圾邮件。

修复方案

在您的 DNS 服务提供商处，为您的域名添加一条 TXT 记录：

  • 将主机字段（Host）设置为您子域名的名称。（例如，如果您的电子邮件地址是contact@mail.example.com，则为 mail。）如果不使用子域名，则将其设为@。

  • 用您的 SPF 记录填写 TXT 值字段。例如 v = spf1 a mx include：secureserver.net〜all。

目前已经越来越多的收件人服务器要验证发送者域名的 SPF 设置。也就是发送的 Ip 或服务器必须在 SPF 中有记录才接受邮件。

首先你必须有自己的真实顶级域名。没有的话是不可能设置SPF 的.
SPF 是域名的一条 TXT 解析记录。

SPF 设置说明：

\1. 把《速递专家》安装到服务器或VPS上面
\2. 域名的解析设置里边添加一条TXT解析

主机名一般为：@，解析值为：v=spf1 ip4:你的服务器IP ~all

例1：假设你的服务器 IP为 123.123.123.123，则 SPF 记录值为

  1. v=spf1 ip4:123.123.123.123 ~all

例2：如果有多个IP可以并排书写，例如：

  1. v=spf1 ip4:123.123.123.123 ip4:123.123.123.124 ~all

例3：如果有很多连续地址，比如 123.123.123.1 到 123.123.123.255，则可以解析

  1. v=spf1 ip4:123.123.123.0/24 ~all

go-开启go module 

1
2
λ go mod init
go: modules disabled inside GOPATH/src by GO111MODULE=auto; see 'go help modules'

开启go module：

1
2
set GO111MODULE=on    //windows
export GO111MODULE=on //linux

上面的命令开启go module是临时开启

永久开启

1
2
3
4
配置powershell
# Enable the go modules feature
$env:GO111MODULE="on"  //windows
#echo 'export GO111MODULE=on' >> /etc/profile //linux

go-gin 常见问题解决

安装过程中遇到的问题

  • cannot find package “github.com/go-playground/validator/v10”

    发现github上确实没有该代码

    • 解决方法：
    1
    2
    3
    4
    5
    # cd $GOPATH
    # go get "gopkg.in/go-playground/validator.v10"
    # cp -rf $GOPATH/src/gopkg.in/go-playground/validator.v10/*   $GOPATH/src/vendor/github.com/go-playground/validator/v10/
    # govendor add +external
    复制代码

  • cannot find package “golang.org/x/sys/unix”

    • 解决方法：
    1
    2
    3
    4
    # mkdir -p $GOPATH/src/golang.org/x
    # cd $GOPATH/src/golang.org/x
    # git clone https://github.com/golang/sys.git
    # govendor add +external